Data Loss and Recovery in the Age of Paper

On the night of November 8, 1800, a fire that began in a neighboring building destroyed the papers, records, and books of the War Department office in the new city of Washington. Within days, secretary of war Samuel Dexter began the recovery process by writing to division heads across the department seeking copies of the most recent records, requests, and receipts so that the business of the department could continue.¹Letter, Samuel Dexter to John Pierce, November 10, 1800, Papers of the War Department, 1784–1800, http://wardepartmentpapers.org/document.php?id=41262 For over 200 years, the records of one of the original government departments were unavailable for research and learning.

The early national government was small, but the administrative footprint of the War Department was large. During the 1790s, the secretary of war spent seven of every ten dollars of the federal budget (debt service excepted). The War Department did business with commercial firms and merchants all across the nation; it was the nation’s largest single consumer of fabric, clothing, shoes, food, medicine, building materials, and weapons of all kinds. Additionally, the War Department operated the nation’s only federal social welfare program, providing veterans’ benefits (including payments to widows and orphans) to more than 4,000 persons. It also provided internal security, governance, and diplomacy on what was known as the American frontier, and it was the instrument that shaped relations with Native Americans. Without access to this store of records, historians had always accepted that there were gaping holes in the records that could help answer a variety of questions about the early American republic.

Bringing that evidence together, which represents the work of early American federal officials, increases access for researching papers that live in the public domain. In 1942, Elbert L. Huber described how the newly accessioned War Department records to the National Archives— without the destroyed documents—held great value because of their aggregation:

“The concentration of these records, once scattered in buildings all over Washington and neighboring depositories in Maryland or Virginia and in some instances in places much more remote, affords the student of military history opportunities in research which have hitherto been denied.”²Elbert L. Huber, “War Department Records in the National Archives,” Military Affairs 6, no. 4 (Winter, 1942): 247–254.

In an effort to aggregate paper copies of records scattered around the world, historian and documentary editor Ted Crackel began a painstaking process in 1993 to reconstitute the missing papers. During the next decade, he and a staff of researchers at East Stroudsburg University in Pennsylvania visited nearly 200 repositories and consulted more than 3,000 public and private collections in the United States, Canada, England, France, and Scotland. Documenting and digitizing, Crackel aggregated copies of nearly 43,000 lost documents and transferred the digital copies to the Roy Rosenzweig Center for History and New Media (RRCHNM) at George Mason University where, with funding from the National Endowment for the Humanities (NEH) and the National Historical Publications and Records Commission (NHPRC), those collections transformed into the Papers of the War Department 1784–1800 (PWD) website, <http://wardepartmentpapers.org/>, a digital documentary edition. PWD represents a digital aggregation of records that do not exist together in a single physical archive, and would not have been possible without funding from federal grant programs.

Now aggregated, researchers can search through the carefully described and tagged documents that help illuminate topics such as relations between Native American nations and the federal government, the care of Revolutionary War veterans and their families, the nature of communications in a sprawling new nation, and the debates over US involvement in global affairs. While military and diplomatic history enjoys a wide audience of enthusiasts, interested citizens, and active and retired military members, PWD also contains easy-to-find names, events, and places, making it a very important resource for family researchers and genealogists.

Not all lost federal records can be reconstituted or reconstructed. Some losses of federal records have been devastating. In 1973, 16–18 million Official Military Personnel Files (OMPF) were lost in a facility fire in St. Louis, Missouri, that will never be recovered. While some advocate for digitizing public records to prevent against potential data loss from natural disasters and accidents like fires, all digital assets must be cared for in specific ways to ensure for their long-term preservation as well. Through a new grant from the American Council of Learned Societies, RRCHNM will migrate the existing digital assets of PWD, originally built with a customized infrastructure, to the standards-based Omeka S platform, which offers pathways for digital preservation and management with Mason’s institutional repository. The Library of Congress maintains guidelines, tutorials, and tools to help federal agencies, and any organization or individual, maintain their digital data. Without following careful procedures for formatting, transferring, and saving digital data now, government agencies may be in danger of dealing with losses of millions of records important for America citizens that cannot “easily” be reconstituted, even after 200 years.