Yesterday at 3:00 PM, the core distribution switch at our Dallas branch office committed hardware suicide. There was no graceful degradation or helpful SNMP trap; the power supply just violently detonated. The local office manager called to report that the entire site was offline and the network closet smelled heavily of ozone and burnt plastic.
The burn-rate timer hit $1,850 before we even got the replacement put in a FedEx box.
Here is what actually happened.
Because the site was hard-down, I immediately called our vendor’s enterprise support line to trigger an emergency RMA (Return Merchandise Authorization). We pay a premium for a 4-hour hardware replacement SLASLAA metric we are currently failing, but will creatively report as 'green' by redefining the outage., so I expected this to be a three-minute phone call.
Instead, I was taken hostage by an offshore Tier-1 support technician and his completely inflexible runbook.
I explained the situation clearly: "The chassis is dead. The power supply is physically melted. It is a brick."
The technician paused, consulted his script, and replied, "I understand you are experiencing degraded performance. Before I can escalate this or process an RMA, I need you to plug a console cable into the switch and provide the output of a show tech-support command."
I spent the next forty-five minutes on the phone, politely trying to explain the fundamental laws of thermodynamics to the vendor. I explained that a blue console cable cannot extract text files from a piece of metal that currently has zero electricity flowing through it.
He refused to budge. He stated that without the diagnostic log file, he lacked the actionableActionableA buzzword used to reject a perfectly good report because the boss didn't want to read it. data required by the system to prove the switch was actually broken. He suggested I might have a "Layer 8 issue."
My manager was forcefully dragged onto the Webex to yell at the technician. We were now burning two senior engineering salaries to debate the physics of electricity with a script-reader. It took a full two hours and three different management escalations across two time zones just to get a piece of replacement hardware put on a truck.
We didn't fix a complex routing loop yesterday. We just spent nearly two thousand dollars in payroll trying to convince an enterprise vendor that a melted power supply doesn't generate log files.
Total waste generated: $1,850.
Next time Tier-1 support asks you to console into a pile of melted silicon, don't argue with them. Just put the phone on speaker, sit back, and start the timer.
