Back in March I wondered out loud whether anyone could be stupider than the IRS with handling emails. Then I learned that Ms. Clinton ran her own computer system for her official emails. One of the many problems the Secretary of State had with running a private mail server is that she or people corresponding with her may have inadvertently leaked sensitive government information. Since our government threw the book at General Petraeus over leaking sensitive government information, it was highly likely that Ms. Clinton would eventually have to pay for her bad decision making. This week the Wall Street Journal wrote:
Mrs. Clinton took an enormous risk to national security by putting her official emails on a private server. Sooner or later she was certain to send or receive some information useful to foreign governments even if it wasn’t officially classified. Every intelligence expert we’ve talked to says it is close to a certainty that some foreign intelligence agency was able to hack her emails while she was America’s chief diplomat.
Now the Washington Times reports:
The U.S. intelligence community is bracing for the possibility that former Secretary of State Hillary Rodham Clinton’s private email account contains hundreds of revelations of classified information from spy agencies and is taking steps to contain any damage to national security, according to documents and interviews Thursday.
The top lawmakers on the House and Senate intelligence committee have been notified in recent days that the extent of classified information on Mrs. Clinton’s private email server was likely far more extensive than the four emails publicly acknowledged last week as containing some sensitive spy agency secrets.
So now I wonder whether she will be offered a Petraeus-like agreement by federal prosecutors in which she would plead guilty to a misdemeanor charge for mishandling classified information. It sounds like her email account contains security breaches that are more numerous and serious than General Petraeus. At some point we have to admit that her dumb decisions with her email account will land her in court, seriously damaged national security, and disqualifies her as a Presidential candidate.
I am not surprised that Ms. Lerner’s emails were found. As an old IT guy who is very familiar with the email system used by the IRS I found it almost impossible to believe that the IRS was so inept with implementing their email backup policies that they could not retrieve her emails. They do not pay IT guys enough to take on that risk without direct orders from upper management. In fact several people familiar with the IRS email systems said it was unlikely the backup tapes were permanently lost because government IT systems almost never destroy or re-use backup tapes. They just buy more. My solution to finding the missing tapes was to arrest the IT guys and see if that improves their memory. Fortunately they did not have to resort to that since Ms. Lerner’s email records were “right where you would expect them to be”. If this report is true, then the question becomes when did the IRS leadership know of the cover-up, what did they do about it, and why did the IRS deliberately impede an ongoing investigation. If the cover-up stretches up to the White House this could be another case like the Watergate scandal where the cover-up is worse than the crime.
In fact, Camus stated, Lerner’s e-mail records were “right where you would expect them to be.” A year ago, the IRS claimed after two months of looking that the e-mail records were irretrievably lost; it took TIGTA just two weeks to find them in the usual backup records. The rather obvious conclusion is that the IRS didn’t want to find those records, and hoped that announcing them as destroyed would end the probe into Lerner’s records. If that’s the case, then it raises the question about just how much IRS leadership knows about Lerner’s communications, and how much they might be involved in them (worth noting: Koskinen came in afterward).
For old IT guys like me the interesting question is why wasn’t most of Ms. Lerner’s emails online? If you have the last three to five years of emails online then a disk crash becomes a minor issue. I get the impression that very few of her emails were online. To an IT guy or gal the difference between online and offline storage is how fast you recover from an event like a disk crash or theft. Online storage also lowers the cost of email discovery. In the vast majority of scenarios online storage is the low cost solution and offline storage is the high cost solution. So why did the IRS choose the high cost solution? The letter from Leonard Oursler, IRS, to Senators Wyden and Hatch provides their explanation but in turn opens the IRS to second guessing. According to the letter “the average individual employee’s mailbox is limited to 500 megabytes, which translates to approximately 6,000 emails”. Prior to 2011 the limit was lower. To support Ms. Lerner’s 67,000 emails from 2009 to 2011 it would have required about 5.3 gigabytes of online storage or approximately .003% of the 170 terabytes the IRS said they had available for online email storage. If management wanted they could have solved this problem by instructing the Exchange administrator to set her quota much higher. The cost for this solution is zero.
To put the email storage requirement into historical perspective here is an announcement from 2007 where Yahoo announced that it was offering unlimited email storage. I believe that Hotmail also offered unlimited email in 2007 but I am little fuzzy on whether Google offered 5 GB or 15 GB for their free email accounts. So if Yahoo, Hotmail, and Google can offer email storage greater than 5 gigabytes for free, what is the problem with the IRS?
First I looked at possible Exchange 2007 restrictions. Harold Wong says in this post that Exchange 2007 supported an unlimited database size. It doesn’t look like the software was the problem.
Then I looked at the cost of disk space. At work we use HP Proliant servers with SCSI drives which I assume the IRS is using, too. In 2009 we purchased a 300 gigabyte drive for about $230. If they were running servers with smaller disk drives then It would be easy to swap out drives and expand the database. In this scenario it would have cost the IRS about $7 to put her 5.3 gigabytes online. At this cost level a compliant email retention policy with 5 to 50 year retention schedule for staff personnel looks like a no brainer. If we go to the other extreme and accept the estimate provided by the IRS that “it would cost well over ten million dollars to upgrade the IRS information technology infrastructure in order to save and store all email ever sent or received by the approximately 90,000 current IRS employees”, you have to recognize that the IRS has already spent more than that on this investigation then they would have by upgrading the email system. The IRS policy to limit the employee’s mailbox to 500 megabytes was not only a costly mistake but it was not even a good faith attempt to comply with the federal email retention requirements. One hard disk drive crash could be explained as bad luck. When you have seven hard disk crashes and you are unable to reliably produce the emails, you have just made an argument for a massive policy failure. Why did they do it?
I was reading the letter from Leonard Oursler, IRS, to Senators Wyden and Hatch and wondered where was the printed copy of Lois Lerner’s emails from 2009 through 2011. Here is what the document says on page 3:
In addition, if an email qualifies as an official record, per IRS policy, the email must be printed and placed in the appropriate file by the employee.
If Ms. Lerner was following IRS procedures then she should have printed a copy of every email that qualified as an official record whenever she moved it from the online storage to the archive. I am guessing that this is the IRS version of a redundant system backup.
Here is another question I have from the document. An email in this document dated June 13, 2011, says her disk drive crashed. If the emails from 2009 through 2011 are lost that implies that she archived them. Why did she archive the 2009 through 2011 emails so quickly? The only answer I can come up with is that she was automatically archiving the emails that are older than a month or two. If she followed this practice than it is unlikely she ever complied with the IRS procedures for printed copies and it explains why electronic discovery is so arduous since the online system contains only the last couple of months of emails. The IRS appears to have deliberately chosen an email retention policy that was prone to both error and abuse. Now they are complaining that searching for Lois Lerner’s emails is so difficult and expensive. I find their whining in the letter to be entertaining especially the part where they complain that more “than 250 IRS employees have spent over 120,000 hours on compliance”. All of these problems disappear if the IRS had chosen to operate a compliant email system in 2011. They have enough money for bonuses to IRS employees who are delinquent on their taxes but not enough money to meet email compliance requirements required by law. Maybe we should postpone all bonuses until they get compliant and fix the Ms. Lerner email problem. They own these problems.
This leads me to my last question. The Inspector General and GAO should have flagged this system as non-compliant. From an electronic discovery standpoint, this email system was born to fail. It is such a bad system you have to wonder why did the Inspector General and GAO look the other way?
For those into conspiracies you have to wonder whether Ms. Lerner knew that the IRS flagged groups with titles including “tea party,” “patriot,” and “9/12 project” for deeper review prior to June 29, 2011 and would likely cause a serious political scandal. This is pretty darn close to June 13 and in hindsight it looks like a very convenient time for a disk crash if you want to cover your tracks.
The missing Lois Lerner emails is a hot subject. There are over 900 comments on the Washington Examiner article, Lois Lerner on IRS hard drive crash: ‘Sometimes stuff just happens’ . Obviously there are a lot of people annoyed with this excuse. The idea that the emails were lost in a disk drive crash rankles me so I joined the fray. I can understand where she may not have lost some of her Word or Excel documents but not her emails. Here is what I said:
To an old IT guy like me, recovering files is a completely different problem than recovering emails. It is my understanding that the IRS uses Outlook with Exchange servers as their email system. When you install Outlook on a new drive it creates a new copy of your mailbox with all of your emails. I literally have done this dozens of times. If she is missing files then we are probably talking about Excel and Word documents that were not emailed, saved on the server, or backed up by some other means. If there are missing emails in her mailbox then she deliberately deleted them and the IRS will have to find copies of them in the backups and archives.
Eventually the IRS and the Administration are going to have a Nixon moment in which they figure out that the cover-up is having bigger political repercussions than the original crime.
As an old IT guy I am slightly amused with the unfolding Lerner email scandal since I thought we had fixed these problems in the last century and the Inspector General did not catch this. Not surprisingly a PJMedia post has a former IRS IT specialist conveying doubt on this scenario. He confirms my guess that the IRS is using Microsoft’s Outlook/Exchange for email and that they have followed the generally accepted business practices for backing up and archiving the emails. For almost a decade I supported Microsoft’s Outlook/Exchange at our business so I have to conclude that if the emails are missing from her copy of Outlook then she deliberately deleted them to avoid prosecution. If they are missing from the backups and archives, she had help from someone in the IT department.
Here is what the former IRS IT specialist said in the post at PJMedia.
First, he points to the United States Code for government record retention. That code, 44 U.S.C. Chapter 33, governs what a government record is and requires that agencies must notify the Archivist of any records that are destroyed and the reasons for destroying them. The code was put into place after Iran-Contra to keep government workers and contractors from deleting records.
Section § 3309 states that records “pertaining to claims and demands by or against the Government of the United States or to accounts in which the Government of the United States is concerned, either as debtor or creditor, may not be disposed of by the head of an agency under authorization granted under this chapter, until the claims, demands, and accounts have been settled and adjusted in the General Accounting Office, except upon the written approval of the Comptroller General of the United States.”
“These environments were required by federal regulations to be redundant and recoverable,” the former IRS IT worker says. “The recoverability requirements were put into place for exactly the reasons we see today.” Disposal of records outside the statutory standards requires permission in writing.
He says that the IRS uses Microsoft Outlook/Exchange systems, which are backed up using Symantec NetBackup.
He also says that “the IRS is the cash cow of the federal government. When they ask for funding for anything it was granted without discussion.”
In the case of the prime contract and record retention, “The IRS IT projects were fully funded and never lacked for resources. To state ‘Backup tapes were reused after some short period’ is a complete joke. The IRS had thousands and thousands of tapes and ‘Virtual Tape Libraries’ (VTL or non-tape backups based on hard drive storage technologies). There was never a reason to reuse tapes.”
Indeed, the U.S. government has been getting out of the tape backup regime for years. The former IRS IT worker points to this ExaGrid document from 2011. In the document, ExaGrid discusses its work with the federal government to eliminate tape backups in favor of faster and more secure record retention systems.
ExaGrid specializes in disk-based records retention systems.
The former IRS IT worker adds that in his time on the prime contract, “I have worked for many federal agencies and the IRS had some of the best people.”
“This reason is why I scoff at the story being put out. Those folks would not have had such a short retention period for email unless they had it in writing from the highest levels. It would have made the local IT water cooler gossip if the IRS had screwed up and lost tons of email by accident.”
Yet the IRS claims that it lost the emails a year ago, and is only now telling congressional investigators.