Intel: Backup Tapes for AMD Trial Were Destroyed in EC Investigation

In its final report to Special Master Vincent Poppiti, appointed to handle the discovery of materials relevant to the AMD v. Intel antitrust case, Intel admitted its IT department in Munich had lost track of, and inadvertently wrote over, backup tapes whose contents may have been vital to the case. The reason why, Intel said, relates to a separate antitrust investigation of Intel by the European Commission.

According to Intel's report, on July 12, 2005, its Munich office was paid a visit by investigators from the EC and the German Bundeskartellamt antitrust agency, in connection with a joint investigation unconnected to the AMD trial. There, tapes were retrieved from storage, sealed in metal boxes, bound with security tape, and handed to investigators.

The Munich office received the tapes back just two days later, along with a directive from the EC's lead investigator that they "are released for unlimited use by Intel and that no further restrictions of use apply." Unfortunately, the employee took that directive too literally, not knowing that the storage area from which the tapes were originally retrieved was being reserved for shipment to the US, for discovery in the AMD case. He then placed the tapes back in Intel's recycle pile, where they were inevitably reused, Intel's report admitted.

Later, tapes from that pile turned up in England, after a format change for backup tapes in Munich led to older-format tapes being shipped to Intel's UK office, where the older backup format was still in use.

While AMD had earlier claimed that Munich's error may have led to the loss of as much as 70% of the data it was seeking there, Intel's report states the loss only impacted 96 of the total selected custodians whose records were to be retained for future discovery - about 9% of the custodians. However, Intel said, as many as 46 of those custodians may have complete backups of their own personal hard drives or other weekly backup procedures; and in the later selection process, only 18 of the 96 may actually need to provide data anyway. So quite possibly, the report implies, no data was lost for Munich at all.

Intel also admitted that weekly backup tape data - literally a backup of the backup - was not successfully migrated as planned between some servers during October 2005, on account of human error. As a result, e-mail records for as many as 121 employees may be incomplete.

However, just after the case was brought, Intel says, its IT managers issued orders to employees to back up their Microsoft Outlook e-mails to newly created "personal folders" (if you're familiar with Outlook, you know about .PST files). This was necessary in order for employees to avoid having their mail auto-deleted after a certain period - a process which apparently could not be reversed by administrators using group policy, or at least was not done if it were possible.

One set of instructions to employees read, "For your convenience, you may find it helpful to create additional e-mail folders to store e-mails. Please reference your Outlook Help files on creation and maintenance of these new folders or contact an appropriate IT person to assist you. Below we have added a 'how to' create new data files guide."

On the one hand, material that was inadvertently lost during the exchange of backup tapes with the EC, or during the migration of backup data between servers, may yet be attainable through a forensic study of these .PST backups. On the other hand, however, some employees may have actually lost their own e-mails during the creation of these .PST files, though they may eventually be recovered anyway from server backups.

As part of its proposed remediation program, pending approval by Judge Poppiti, Intel has now purchased an EMC Centera e-mail archiving system, with accompanying EMC servers. That system will be used to re-capture Exchange-based e-mail from all 1,023 custodians, including the 96 whose e-mails were impacted by the Munich incident, and the 121 for whom the redundant backup process was inadvertently overlooked.

Intel's report describes this new backup system as follows: "In particular, the Exchange Servers that contain accounts for all of the currently employed custodians feed into a primary and back-up journaling server. The journaling server, in turn, writes off all sent and received e-mails of the designated custodians to the EMC Extender Server, which is replicated on a second EMC Extender Server as a backup. The EMC Extender Server then writes the e-mail to an EMC Centera Server, which is set up as a write once-read only storage device. A second EMC Sever has been set up and replicates all of the first EMC Centera Sever as back up to the system."

By backing up the Exchange files currently online, dating back to the time each custodian's entry into the Exchange journaling system was launched, all non-system e-mails could be re-indexed by EMC's content addressed storage (CAS) software, which could end up making discovery easier anyway. Some may wonder why such a system didn't exist to begin with.

Intel paints a picture of a company attempting to comply with a handful of simultaneous orders, and in the process of collecting what could wind up to be terabytes of discoverable data, dropping a few crumbs along the way. As Intel spokesperson Chuck Mulloy told BetaNews this afternoon, "We continue to believe that the human errors that led to the document retention lapses will not materially affect AMD's ability to press their case. Intel continues to deny AMD's antitrust allegations."

AMD is reserving comment on the Intel report pending more extensive review, including the resolution of a pending argument over whether communications between Intel employees and/or executives and company counsel is privileged and exempt from discovery.

When Centera servers were released in 2003, EMC's suggested sales price was $64,000 for a 4 TB hardware configuration, plus $84,000 for its accompanying software. For those who are wondering, Centeras are based on Xeon processors.