Four-year-old Diebold bug bites a California election

An unusual project monitoring the accuracy of electronic voting in Humboldt County, Calif. uncovered a glitch that dropped or miscounted 197 ballots in the November election. The bug has been known to the manufacturer for four years.

E-voting's troubles are widely known by now, so much so that Humboldt County registrar Carolyn Crnich consented to work with a group of election integrity experts -- banded together and calling themselves the Humboldt County Election Transparency Project (HCETP) -- to find a way to double-check the results of the June 2008 voting in her county. Humboldt uses Diebold optical-scan units running, as it turned out, version 1.18.19.0 of the firm's Gems software.

Note the version number. Someone at Premier Election Solutions (which changed its name from Diebold in August 2007) did, in an e-mail sent to Lindsey McWilliams, Crnich's predecessor, on October 19, 2004. That e-mail detailed a fix for that version of the Gems software. What it fixed wasn't made clear -- the memo only said that "an issue exists with correctly sorting committed decks, in some reports, and also deleting other decks under certain conditions, when 'deck 0' has not been deleted."

What it meant, for those who don't speak Diebold, is that the machines using version 1.18.19 (and the later 1.18.22 version) were dropping votes.

Political careers being what they are, Crnich didn't see that memo until last month, after local programmer Mitch Trachtenberg uncovered the 197-vote gap. (According to political commentator and talk radio host Brad Friedman, writing in The Brad Blog, California's secretary of state also didn't receive a copy of the memo, despite her "top-to-bottom" review of all California e-voting machines in 2007. A Diebold spokesman describes the 2004 memo as "a workaround.")

Trachtenberg, one of the HCETP group who approached Crnich with their concerns, pulled together an open-source method of scrutinizing a closed-system problem. He describes the process and his tools on his site -- start with Debian Linux, slap SANE (Scanner Access Now Easy) around a bit to get it working with his Fujitsu scanner, and get ready to re-scan and re-tabulate the November ballots. (The group did a test run on the project with June's results.)

At that point, Trachtenberg had to take particular care not to risk invalidating the entire batch of ballots. If a ballot can be identified, the vote can be sold, so California law invalidates any ballot that can be traced to an individual. While he awaited their arrival, he wrote a program that could scrub any potentially identifying information from each card. Additional modules were written to increase project transparency, and the project was off to the races -- deputized volunteers, chain-of-custody controls, and all.

Of the 64,161 ballots cast in Humboldt County in November, 197 were found to have been dropped -- about .0031%, in other words. (Could be worse; county officials say that no race outcomes were affected.) Trachtenberg's full story of the discovery is a must-read, especially if you've ever been on a project that made you feel like you were at a parade honoring the emperor's new clothes.

Now what? Premier says that the "product advisory process" has been changed since 2004 to ensure that secretaries of state hear about issues with software or hardware as soon as the notices go to the actual election precincts. The company rep also said that software recalls and re-issuance of manuals weren't practical options, federal and state certification processes being already expensive and tedious affairs.

In Humboldt County, Carolyn Crnich ended up re-certifying the November elections in light of the corrected results. And Mitch Trachtenberg's got a great idea for a new business -- with his Ballot Browser software a manifest success, he's launching a new turnkey vote-confirmation solution early next year as Trachtenberg Election Verification Systems. The project proves that independent counts are practical and safe -- and that without oversight errors happen: "Our votes are too important to be counted by secret software running on black-box vendor machines."