In 1972 I was the Development Manager at the IBM Program Information Department, called PID.  This department was responsible for distributing software products to U.S. Customers and to sister Program Libraries in Tokyo, Paris, Toronto, and Rio de Janeiro.  My boss, Marty Kloomok called me early one morning to tell me the place was burning down.  The following is a copy of a brochure we published after we recovered from the fire.  To view the photos in full size click on the photo thumbnails and then to return to the article click on your browser back button.

IBM: An IBM user

The IBM Program Information Department fire of September 1972 made headlines.  As an IBM Customer, you quite naturally have far more than a passing interest in that disaster.

Well, we're a data processing installation, too, using IBM equipment... and we learned a great deal from the fire.  The purpose of this folder is to tell you what happened and what we did about it, so that you can benefit from our experience.

The PID resource

Although you are probably familiar with us and our operation, a few words of background may be helpful in putting the event into context,

At the time of the fire, the Program Information Department was located in Hawthorne, New York.

We are now fully operational in Mahwah, New Jersey, about 30 miles from Hawthorne.

Our mission is to distribute all programs, which are available from IBM. Shipments are made to all customers in the United States.  This distribution includes supplying master copies to five other IBM distribution centers around the world.  Thus, PID is a critical resource to IBM and plays a vital role in serving all IBM customers worldwide.

To give you an idea of our activity, on an average day we ship 1,500 program orders and supporting materials to domestic customers.  Each of these orders is accompanied by associated documentation, and is suitably packaged for mailing.  Accordingly, PID is required to store large amounts of program documentation and packaging materials.

To perform our mission, three key elements at PID constantly interrelate.  The first is the product we distribute-copies of our Program Master Library.  Second, our Customer Master Record files, and In Process Order Data.  And finally, our own operating systems, management programs and production programs.

Fire

On Sunday, September 10, 1972, at 12:15 AM, a smoke detector went off alerting the security guard.  Within minutes, he located a fire in a large storage room on the lower level of the building.  The flames were already too extensive to be controlled with hand extinguishers.  The smoke detector also set off an alarm in the nearby fire department, and firemen with equipment arrived on the scene at about 12:25 AM.  Before the fire was extinguished, more than 200 firemen had assisted in fighting a 12-hour holocaust hot enough to buckle steel girders, make plastic butter out of keyboards and tape reels, and reduce tons of paper to ash.

The storage area, where the fire began, contained packaging materials, shipping cartons, cases of 80- and 96-column cards and pallets of program documentation.  Although the packaging materials, cartons, and some cards and manuals were entirely destroyed by the fire, much of the card stock and tightly bound manuals were only charred around the edges; they were ruined by the water and foam used to extinguish the fire.

The storage area also contained copies of the master tape library, kept in steel cabinets within a concrete block vault, protected by a co₂ extinguishing system.  The tapes within the vault were not in any way damaged by the fire, but the bottom tier was soaked by water seeping into the vault.

The computer room, which was located directly above the storage area, received its major damage above the entrance of the air-conditioning ducts from the basement.  At this point, the heat was sufficient to collapse the armor ply floor panels (one inch plywood core sandwiched between two thin gauged steel sheets).  The framing and pedestals also collapsed.  To illustrate the heat intensity under the raised floor, the aluminum caps on the pedestals melted, which would require a temperature of over 1000º F.

Although the computer room had been transformed into a hot oven, the fire damage in this room was restricted to an area directly exposed to flames from the ducts from the basement.  In the flame-affected area, two units were consumed by flame, a 1442 and a 2311. A second 2311 adjacent to the first one was ruined by excessive radiated heat.  A System/3 unit was partially destroyed by fire, which entered the unit from the cable hole openings.  However, the fire did not spread to other sections of the System/3.  Most of the other data processing equipment in the room was damaged, not by actual flames, but by heat and smoke.

Although the room contained a significant number of tapes, some lined up on shelving, in cabinets, or on low flat carts, a large percentage of them appeared to be intact.  There was much more damage to the reels themselves.  Some of the reels high up on the shelves and near the floor (on dollies) showed the most damage.

We had suffered a great loss; our business was heavily impacted.

First steps to recovery

The facilities and the data processing equipment sustained heavy damage, but the real story of the PID fire is in its aftermath ... the recovery.

By 3:00 AM Sunday, the morning of the fire, the IBM PID management team was at the site.  By 4:00 AM, a temporary control center was set up in a nearby motel.  At about noon, the fire was under control, and by 2:00 PM, leased space at Mahwah had been selected as the site to which PID would move temporarily.

By late Monday, two System/360 Model 65s, which were being removed from Mahwah, were reinstalled and operational.  All remaining equipment had arrived and was installed by Thursday afternoon.  During the week, tapes were recovered, machines installed, systems generated, and documentation and supplies ordered and delivered.  Also during the week, the PID assembly and packaging operations were reconstructed.  All our packaging material and shelving had been destroyed and had to be replaced; however, we were able to salvage a conveyor belt system and two large packaging machines.  By Friday afternoon, PID was operational.  On September 18, eight days after the fire, customer orders were being shipped.  And on September 29, nineteen days after the fire, the backlog was clean and PID was completely current.

All this was made possible by two factors-emergency planning and employee dedication.

As soon as the alarm went off in PID, IBM Security was notified.  They, in turn, put into action an emergency plan, which started by notifying all executives with direct control over any resource that might be important for recovery.

Thus, when the PID Management team convened at the motel emergency headquarters early Sunday morning, the necessary IBM people were on-site and ready to help.

The first critical problem was temporary space.  A representative of IBM's Real Estate and Construction Division arrived with an inventory of all available space within commuting distance of Hawthorne.  Fortunately, the Mahwah building, which had recently been vacated by another IBM activity, provided a match to PID's requirements.  Equally fortunate for IBM was the availability of two recently discontinued Model 65s, which were being crated at the site.

Also present at the Sunday morning meeting was a member of the Data Processing Division's Product Scheduling department.  His job was to get the next most critical resource-equipment.  Since space was now available, he could go to work immediately.  The procedures that he followed were exactly the same as those used in the event of a customer emergency.  The installed inventory files were checked to determine what equipment-down to the feature level-would have to be replaced, and expediters were named in headquarters and in each of the involved plants.

In addition to the System/360s, the installation also required System/3s for generating 96-column cards and 5440 disk packs, 1130 computers for generating 2315 disk packs, and a Model 20 with a special RPQ for producing paper tape.

By Monday afternoon, Customer Engineers were reinstalling the Model 65s.  Shipments of the other equipment started arriving on Tuesday and, due to the emergency support of a Field Engineering task force, the PID installation was, for the most part, recreated at Mahwah by Friday evening-five days after the fire.

One other key activity was started at the motel headquarters on Sunday, the morning after the fire.  The divisional Purchasing department was asked to review all PID purchase requisitions and contact the appropriate suppliers.  By Monday afternoon, emergency orders had been placed for all the furniture, equipment, packaging materials and paper forms required by PID to process and ship customer orders.

Recovery

The PID disaster plan is only a three-page document.  All the managers had participated in either drawing up the plan or auditing it, so that each was familiar with the problems he would face after a disaster.  The fact is that the enforced discipline of thinking through the plan prior to an emergency turned out to be more important than the document itself.

With all of the peripheral support machinery put in motion, the PID management team turned its attention to the specifics of the plan for the recapture of the three elements required to resume normal operations:

The Program Master Library
Customer Master Records and In-Process Order Data
PID'S own programming systems

The Program Master Library (almost 2,000 tape reels), which was kept in the basement vault, had one of the best backups of all of the PID resources.  Sub-master (working) copies were kept in the computer library just off the computer room.  Another complete set is kept in Toronto, and a third complete set in Paris for distribution to IBM World Trade Corporation's customers.

All 80-column card programs and paper tape programs (kept on tape), and all System/3 card and disk programs and 1130 disk programs (kept on disks) were recovered on-site.  A considerable number of tape submasters were in process in the computer room and were destroyed by the heat of the fire.  Of these, all but 121 were recovered from the basement vault.  These 121 were in the bottom tier of the vault where the water and foam had seeped in, and rather than take any chances, the backup libraries in Toronto and Paris were used. (Interestingly enough, many of the water-soaked tapes were found to be readable after they had dried out.)  

While the Program Master Library was common to the Canadian and European operations, the Customer Master Record files and PID'S own programming systems are unique.  Consequently, the PID emergency plan called for extensive backup for these files and programs.  The active files and programs were kept on 2316 packs, which were online.  A backup set on magnetic tape was stored in a fireproof vault in the tape library.  Another set was taken each week to an off-site vital records storage area.  Recovery was effected from the tapes stored in the vault.  In the case of complete disaster, recovery could have been effected from the off-site copies.

Once the vital records were recovered, the recovery of the work in-process was the final step.  The assumption was made that all in-process customer orders were destroyed and special programs were written to reset all files by "backing out" in-process data.  All orders were then reprocessed and all customer-owned tapes and disks on the premises were replaced.

Although a large amount of program documentation was salvaged from Hawthorne, there was 100% backup of this documentation in the Publications Distribution Center in Mechanicsburg, Pennsylvania.

It can happen to you

In summary, the major factors in our quick recovery were planning and people: planning to think through the unthinkable and establish backup and recovery procedures; and people, whose training, dedication and commitment are a vital requirement in any emergency.

At PID we have indeed been tested by fire.  We believe that helping you guard against a similar disaster falls squarely within our mission of customer service.  And the experience of the fire enables us to make some specific recommendations.

Computers themselves are not combustible and the supplies that are used to support them are only slightly so.  Our computers were damaged by heat but they never burst into flame.  Cards, tapes, disks and documentation in the machine room did not ignite.  Cards stored right next to burning material charred on the outside but did not burn.  The recommendation is to store combustibles in an area far removed from computers and supplies, preferably off-site.

The effectiveness of fire walls in any facility was dramatically illustrated.

And, perhaps most important, have a plan for recovery.  In an emergency, you may not be fortunate enough to have the planners around to implement it, so the document should be stored off-site.  It should contain an inventory of items essential to recovery, a list of alternate sources of supply, and people to contact at the supplier.  The plan should be simple and flexible.  It is more important that the plan identify alternatives than give detailed instructions.

A manual, "The Considerations of Physical Security in a Computer Environment," can provide you with more information on the subject.  It is available from your IBM representative.

IBM  

International Business Machines Corporation

Data Processing Division

1133 Westchester Avenue, White Plains, N. Y 10604

(USA only)

IBM World Trade Corporation

821 United Nations Plaza, New York, N. Y 10017

(International)