Basic oversights create high-tech havoc
Sometimes in IT classic "d'oh!" moments sneak up on you. This particular situation occurred sometime in the mid-1980s, back when the Web was in its infancy or maybe even before it was conceived.
Posted by Anonymous on January 13, 2009 03:00 AM on InfoWorld
I worked for a large corporation on a new project that involved shopping kiosks that one would use for purchasing goods from a number of recognized merchants -- a project considered quite high-tech at the time. The terminals featured a touchscreen, keyboard, credit card reader, and receipt printer for the transaction. In addition, it had lots of color images of products and an interactive touchscreen interface to make shopping for items on a computer more like shopping for real. We placed terminals in shopping malls and areas where there would be lots of foot traffic. In addition, we placed a terminal on the floor in our office so that we could use the system ourselves.
As part of the pilot, we distributed about 40 of these terminals around the local metropolitan area to introduce the public to the kiosk's concept. I was a systems programmer and was responsible for the communications code that enabled price changes, sales information, and other data to be transferred to and from the mainframe computer. The protocol we established was that the kiosk would collect sales during the day, and at a configured time it would place a call (no TCP/IP) via an internal modem to the datacenter and upload the day's sales. Next, it would download from the host any price changes, identities of items to be removed, and so on. Finally, it would obtain from the host the next time it should dial in for data exchange and the phone number for it to call.
One day, we had to make a change to the communication software so we sent a programmer to the datacenter to install the change and test it. Later that afternoon, this programmer and I were hanging around the office of the CICS programmer and someone walked up and told us that the kiosk on our floor was constantly dialing. She was a project member and was able to obtain the phone number it was attempting to dial. When she told us what the phone number was, the CICS programmer reacted.
"That's my realtor's number."
We let that sink in for a few seconds. Then he told us that he had used that phone number for every data entry field that required a phone number on the test CICS system. (He was in the process of buying a house at the time and I guess that's the number that was very much on his mind.) When the CICS programmer shared that information, the programmer who earlier had installed the change to the communications code reacted.
"I forgot to switch back to production after testing my code at the datacenter!"
That's when we all realized why the kiosk in our office was constantly dialing: When the kiosk began its communications sequence after the systems programmer ran his test, all the sales information went to the test environment, and more importantly, it was instructed to dial the CICS programmer's realtor's office for the next exchange -- which was set at 4:00 that afternoon. We also realized this: The kiosks were programmed to retry every minute after a failed communications attempt. So every minute it would dial a well-known real-estate office, listen for a modem tone, and when none occurred it would hang up. Then it dawned on us that the 40 other terminals around the area (some up to 2 hours away by car) were doing the same thing. The only way to correct it was to reset the phone number on the kiosks themselves, because once the kiosks had the phone number changed by the process in place, they were effectively cut off. They no longer knew the datacenter numbers, they only knew a bogus number (the real estate office) which wasn't giving them any useful information back.
We called the realtor's office to let them know what was going on, then we resolved the problem by dividing up the area among the project members, driving out, and resetting each machine. The realtor kept staff at work until late that night, answering the calls. The next day we used the kiosk on our floor to send flowers and a note of apology to the realtor's office. I guess they decided they really wanted the sale on the CICS programmer's home, because the realtor didn't pursue any action.
Eventually the project died and the project team was first in line for the fire sale of all the unsold merchandise we had in a local warehouse. I still have the set of screwdrivers and some wood tools from that sale.