A new work week has begun, bringing with it the hope of new achievements — and also the possibility of screwing something up, as happened to IT specialists who talked about their worst moments at work.
A new work week has begun, bringing with it the hope of new achievements — and also the possibility of screwing something up, as happened to IT specialists who talked about their worst moments at work.
The British publication The Register has a column where IT professionals share their failures at work. We’ve selected five stories that almost cost IT professionals their careers.
The first story comes from Sam, whose real name The Register is not releasing. In the late 2000s, the IT specialist created an app for his employer that he described as a «content migration toolkit.» The program worked so well that they decided to commercialize it, but that required a licensing system.
«Excited by the challenge, I spent the weekend researching asymmetric keys and created a licensing system that periodically checked the server, both at startup and at regular intervals,» said the hero of the story.
Over time, we decided to launch new features that required more work. Sam couldn’t finish coding during work hours and took his laptop home to work on over the weekend.
On Monday, Sam’s phone rang on his way to work. It turned out that the licensing program was down and no one could log into the content migration toolkit.
«And then it dawned on me: the licensing server was still running on my laptop,» the IT guy said. What’s more, he realized that he had never moved it to a production server. For years, it had been quietly working on his laptop, doing its job well.
So when Sam arrived at the office, his first job was to deploy the licensing program to the appropriate server.
This story was shared by an IT specialist, whom the publication called Rufus. He once worked for a British retail chain that offered credit accounts to its customers.
To test the website that customers used to manage these accounts and the internal applications, the retailer created test accounts that Rufus and his colleagues could use to conduct fictitious transactions.
These accounts had names like «Mr. Test Account Six,» and their address was always listed as the main office of the retailer where Rufus worked. Transactions made using the test accounts were never processed as real orders. But each of the accounts had a real credit limit.
«We had to remember not to order anything expensive,» Rufus told The Register, because if the test accounts exceeded their credit limit, they couldn’t make any more purchases.
Everything was fine until a letter addressed to «Mr. Test Account Six» landed on Rufus' desk, saying that he was in debt and needed to pay as soon as possible. Then other letters started coming in, which he ignored and just laughed at the fact that «Mr. Test Account Six» was getting letters.
But one day, Mr. Test Account Six’s account mysteriously disappeared from the system. It later became clear that this was due to ignoring payment requests.
As it turned out, the collection agency had bought the debt and the right to collect it. Then the developers realized they should have taken this more seriously, especially since Mr. Six had their office address.
So the tech team’s amusement turned into fear that burly guys with baseball bats might show up at any moment and demand to see Mr. Six.
«Luckily, the credit team settled things with the collection agency,» Rufus recalls. «None of the collectors showed up, and we got a new account for Mr. Six.»
The next hero is Kane, who spoke about the most terrifying experience of his IT career. «If I hadn’t taken my blood pressure medication, I’m sure I wouldn’t have survived,» he said.
Kane worked for a global hardware vendor and was asked to help an account manager sell a large batch of new servers to a customer who was already unhappy with the service: they had been given switches that were causing problems with their virtual LANs. But when the customer was offered to fix the switches, he agreed to consider new servers.
So one Saturday morning, Kane found himself in the office trying to fix a VLAN-related «partial failure» that had brought down a significant portion of the SAP implementation.
«I decided to show the client how to make changes using the configuration web pages. I was never told not to use a GUI, so I didn’t see any problems using it.»
But Kane didn’t know that the graphical interface was buggy.
«I ended up causing a level 2 broadcast storm that took down most of the network,» he admitted. SAP now couldn’t access its external storage and, at first glance, looked completely dead.
«I imagined I had destroyed their ERP system and there was no telling how long it would take to restore it,» Kane said.
Fortunately, the infrastructure on which SAP was running was well-designed and reasonably resilient. It didn’t take much effort to get the network and ERP system back up and running.
But the client was not impressed and went to a competitor.
The so-called David worked at a small company whose computer suite — web systems, database, back office, etc. — fit on a few racks in a server room. But the company grew and this one room became insufficient, so they decided to move everything to a larger one.
Everything needed to be moved to a larger space in the same building. Step one was to build new racks in the larger room, connecting cables and patches for power and network. Step two was to install redundant systems where possible.
Border Gateway Protocol (BGP) was used to ensure that clients were routed to the location that was best for them, so there wasn’t even any downtime needed while these backup systems were being moved.
Then it was time to move the only system that didn’t have any redundancy: the Oracle database that the business relied on. This was done with a short downtime planned—a night during which David and his team could physically move the machine from one room to another.
The plan went smoothly—the system was installed, powered up, and tested just minutes before the downtime window closed.
The tired IT guys were happy and sighed with relief—until David, finally relaxing after a hard night’s work, leaned back against the big power button on the rack. The rack with the database on it. The database, which was only one. On which the entire enterprise depended.
A frenzy of activity ensued as the seconds counted down to the moment when the business would start losing money if all systems were down.
But David couldn’t remember if it had happened on time or maybe a few minutes later. For him, time had stopped.
What he did remember was that a plan was immediately put in place that ensured that the database would also be redundant, and that no single button could ever take the entire company offline again.
Spencer worked for a UK government agency as a «third line support/infrastructure specialist.» That status may seem modest, but it was enough to land him and a colleague on a flight to Glasgow «to carry out various tasks in a fairly secure location.»
Having accomplished everything planned, the IT guys decided to check out the «data center» — actually a small room full of cleaning supplies, stationery, bottled drinks, and a CAT5 patch panel.
«We thought it would be a nice added bonus,» Spencer explains.
Of course, things went wrong. While tidying up the cables, they managed to knock over a fiber-to-copper media converter. The box broke the fiber-optic cable as it fell, meaning that their attempt to clean up the patch panel had caused them to disconnect something. Something that was probably important.
When the men realized what they had done, it became clear that they had chosen the worst cable to cut. It was an ST-LC cable, which is not common. But Spencer and his partner needed one—and quickly.
«Panic set in. It was 6:00 p.m. All the stores were closed and we were far from civilization,» Spencer said.
The minutes ticked by. The situation didn’t look good. «I felt a cold sweat of fear from complete helplessness,» the IT specialist recalled.
And then his colleague cleared his throat.
«I turned around and saw he was holding the Holy Grail cable. I don’t know where he found it, and I didn’t care.» But Spencer was really worried about whether it would work — it did — and whether he would be able to catch his plane home — he did.
If you want to share your own facap (even anonymously), write to [email protected]. And every week we will highlight the most interesting and non-trivial stories about the failures of Ukrainian IT specialists.
