I had a customer who kept their HP file server in the office manager's office. We get a call from her one day - the server is making loud beeping noises and it's slow.I head over, and see that the LED for one of the RAID 5 drives is blinking.

I run the RAID admin app, silence the alarm and check the logs - no errors from that drive prior to it going offline.I pull the drive, and carefully check for damage and dust bunnies on it and it's slot. Seeing no physical issue, I replace the drive. It's detected and immediately, the RAID begins to rebuild.

After about an hour, the rebuild completes without error. I recommend to the office manager that we order a replacement drive, and schedule the replacement for the next weekend. I pack up and go.

A few days later, she calls and says the RAID alarm is sounding again, but this time, the server is not running. I ask what happened.

She said that everything was working when they got in that morning, but that soon after, the alarm started going off. The company president came by to ask about the noise. She told him that the last time it happened, the computer guy pulled out the drive with the blinking light and put it back. He tells her to try that, so she does.

When the alarm doesn't stop, he says, "Well, that didn't work. Try another one!"

Me: "OK, please pack up all the backup tapes and have them and the server ready for me to pick up. I'll need to do a full restoration."

Background: I was vendor support for a service running on one of this company's servers. We had ruled out any issues on our end and pointed at a local network issue, but I was needed for testing at the end so I got to stay as a fly on the wall. This had turned into a big meeting with several network people, sys admins, several managers and upper management, and one lonely local IT guy who was the only set of hands on the affected machine. We had decided to change the IP on the affected machine.

A local IT guy (IT), a network guy (Net), and network guy's manager (NetMGR) walk into a bar...

Net: Alright, I'm going to send you the new IP in the meeting chat. You'll need to set it as a static on the server.

IT: Got it. Entering it now. ... Done.

Net: I'm still not able to reach it... Can you try rebooting it?

One reboot later.

Net: Still nothing... Can you send me a picture of the info you entered into the static just to double check?

IT: Sure, trust but verify. I got you. Give me a second.

one blurry phone picture later, since the machine's offline at this point.

Net: This doesn't make sense... That's exactly what I sent you. Can someone else try to reach it from a different source?

NetMGR: Net, can YOU send a screenshot of what you entered on our end for that port?

Net: I guess? I copied it straight out of the entry...

Net sends a screenshot that has an IP that clearly does NOT match what he sent in chat earlier. I'm talking 10.10.10.10 compared to 10.10.212.47. Not even close. It was probably an IP address from a different entry altogether, but I'll never know.

Complete.

Silence.

(For much longer than was comfortable.)

IT: And... done! Can you reach it now?

Net: Yes...

IT: Cool, can we check to see if the service is working with the new IP?

Me: Uh, yeah. Testing now.

The meeting proceeded and it ended up being something else (still a network issue, just something outside my wheelhouse). I'll never forgot you, super chill local IT guy.

I used to work for a company that made automatic people counting hardware (it's a thing, honest). The last model we developed was a time-of-flight based device that worked by tracking people across virtual 'in' and 'out' lines. It had to be installed at a doorway and counted people as they entered and left. It was capable of greater than 99.5% accuracy, but only if installed correctly.

It also had a built in low res camera making remote setup possible, and we had our own cloud based reporting and remote management service tieing everything together. We started selling the compete system to both our existing partners, and occasionally direct to end customers.

We were acquired by an American corporate about 15 years ago who were squeezing the workforce, making redundancies every single year, and making some ridiculous management hires and decisions. Choosing to effectively compete against our own partners was just one of the ridiculous decisions they made.

With all that said, management jumped on any potential sale regardless of any red flags. One such sale was 5 devices to an Israeli company who installed them in a cinema to use to monitor & enforce social distancing. First red flag was that they insisted they didn't need any software as they were going to integrate the devices into their current system.

A month later they came back to us complaining that the integration is too difficult (it's really not), and that they need a discount on our software because their client doesn't have budget for anything else. Not our problem? Wrong. Management decide they can have our reporting and remote management service for 80% off! They agree but will only pay once it's all working. Again management agree against my better judgement and without any kind of statement saying what actually has to happen for the system to "be working". Anyway I go ahead telling the customer what settings they need to make for their devices to connect into our cloud. Once they do the network stuff, I can then configure everything else remotely.

Getting the devices online takes weeks. After multiple emails, eventually I see the devices connected in. I then start setting things up and a whole bunch of issues become evident. Essentially, four out of the five devices are installed in completely the wrong place. One device was installed in the middle of a wide corridor too wide to cover with just one device, so people were being missed completely. Another device was installed about 3m to the right of a door meaning it would only ever be able to count any one who came smashing in through the window. Two others had issues. Only one was a passable install.

When all these issues were reported back they insisted that it was fine and would work. I reminded them that we had remote access and could see that the install was terrible and could not possibly work. They said it seemed fine and they would monitor it.

About 2 weeks later they come back to us to complain that the Israeli police had been in to check social distancing was being enforced, and the cinema had been fined because the system said there were 50 people in the building when their manual counts revealed it should have said more like 400. We were never told that this was a possibility, but I again explained that the system is not working because of their installation. I can see its not working and, again, they need to move the devices. They go away, this time saying they'll move them.

3 weeks later, they come back again. They have moved the devices and we must set them up again immediately! Geez. Ok let's take a look. I see they've moved only 2 devices. One of which had moved about 2 inches, but the other has moved a couple of feet. Unfortunately they moved the only one of of the five that was previously in a good position. I explain that they've screwed up, not done what they said they would do, and that it still will not possibly work how they have left it. That one unit is still counting nothing unless people jump through the window.

They now insist that I come to Israel to put it right. Management think this is a great idea. I firmly state that there is absolutely no way I'm going over there to help this customer, and we have remote access so can see the issues. The customer is clearly incompetent and/or lazy and they just want us to do what they could perfectly well do themselves. Management thankfully agree.

After yet more emails and conference calls, they say they'll move the devices.

A week later, "the devices are moved, please check". The devices are not moved. That one device 3m away from the door has been tilted sideways towards the door, and I can see they have installed one of those 'tensa barrier' things to guide people towards the devices field of view.

I again state this is not good enough. They might have improved it a little but it's not fixed it properly. They say they'll monitor it again.

Unfortunately the next week the Israeli police are back and the cinema is fined again. They demand that we fix it. I tell them again, that it is their responsibility to install the devices correctly. Again I tell them I can see the install is bad.

At this point I compile a spreadsheet of time spent supporting them versus the sale cost and I send it to the management team. It turns out we made a loss of about £1200, and I point out that we're still supporting them and they've still not paid for the software. I'm then authorised to tell the customer that they are no longer a partner and we can no longer support them.

It's one of the most satisfying emails I've ever written.

While still living in Pennsylvania, we had given my mom my SO's old macbook pro when we upgraded.

We moved off to Seattle. That's a 3 hour time difference.

One morning, we were both getting dressed for work, when she called. Abnormal because no one calls me unless someone is in the hospital (some time not even then), dead (some time not even then), or in the hospital (some time not even then).

For the life of me I cannot recall what she said was happening with the computer.

I'm trying to probe information out of her while getting dressed, on the way to work, and as I walked down the hall to the office door.

Then she says something that makes me ask: Are you still using the Macbook?

Her: No, it died awhile back.

Me: having died inside and joined the Macbook in the 8th level of Hell Mom, how were you following my instructions?

Her: I know you're a macperson but...

Me: That has nothing to do with anything. They're very different in...

Her: Fine I'll never ask you for help again

And she never has.

Recently, I was taking holidays with family. It was my last night in the USA before coming back to France. I was dinning with some friends of friends, and I overheard somebody complaining about a technical issue. It seems it was really impeding some personal project she had.

I'm at ease with computer, and that seemed fixable in 3 minutes. Admittedly, I understand it was not intuitive, but I asked her to bring her laptop so we can work on it. She didn't have it, which makes perfect sense. So I suggested we meet tomorrow morning, as I was leaving at noon and would have plenty of time to deal with it. I was staying at only 15 minutes by car from her place (I'm from Paris, France, I can't imagine a town does not have an acceptable bus service, but I guess she had to drive, I was trying to adapt myself to this culture).

Anyway, she told me that she does not want to bother me during my last day of holiday, and she prefer to call me once I'm back home, at my place.

I tried to explain that it would be far less bothersome to take a laptop and do the thing myself than try to explain to her how to share her screen and follow my instruction. To no avail. I wonder whether it's because before noon is to early on a Saturday. I explained plainly that I would not do it over phone, and it's either while I'm here or it's somebody else problem.

I told my friend not to give her my contact. I wonder whether she'll still try to get help or not. Until then, I guess it's "tales from no support", sorry

This is a story about a tech support guy from a large multi-system vendor. We'll call him Sherlock.

We have about 100 blade enclosures from this vendor, each one loaded with hypervisor servers that themselves can have a couple hundred guests a piece on them.

Now I'm the manager of a NOC, and my analyst that was working this issue was about a month into a new job. We'll call him Steve.

Steve has been trying to get Sherlock to replace what we knew was a faulty IO module in this chassis. But Sherlock has a script to follow, and no matter how hard our module fails to work properly, he's not replacing that thing until he's done with his script.

I can only figure he got to the section that says "Warning: non-production troubleshooting only!" And tells Steve "Hey it's time to reboot the chassis"

Now Steve's bullshit meter goes off, and he turns to me and asks me how we reboot a chassis. I tell him we don't, what maniac is asking for that?

So I have Steve conference me in, I ask for a quick recap, and then I ask Sherlock if he asks people to reboot a whole chassis often.

No, not really he says. Most people refuse.

So then I ask him how often he has people reboot a whole chassis for a SINGLE BAD PORT on an IO module?

".... I'll start the RMA"

I don't know if Sherlock was just having a bad day, but Steve asked him 3 times before me "is this really necessary for a bad port?"

For context, rebooting a chassis would have immediately hard dropped 9 blades, and roughly 400 production servers, without any change record in place. I have plenty of capacity to HA these to other hosts, but I need approvals for all of that. All for a bad network port.

I got a phone call while at work on Tuesday, my Aunt's computer wasn't connecting to the internet, I said call back the next day when i'm not at work.

At about 8am the next day (Wednesday), while very asleep, I got rang up, after a quick bit of troubleshooting it appeared her wireless card was disconnected or broken. They offered to do a video call to show me the PC. I talked them through uninstalling the network drivers in Device Manager and restarting the PC and i'd call them back shortly.

After doing my morning routine of cleaning myself, getting dressed and using the loo I called my Aunt back. She showed me the PC and there was no wireless car d to be found and nothing on the PC indicating she had one, she was however connected to her old router for a provider she stopped using a year ago. I realised fairly quickly what had happened as I'd seen and heard about it before on more than one occasion.
She'd not unplugged anything when she cancelled and the provider was slow about cancelling so she'd had a year of connection to her old provider before they got around to cancelling it. I bought a USB wireless card online and had it sent to her house. It arrived on Thursday but she was busy and would have to deal with it today (Friday). I sent her a brief message advising her to plug it in, let it setup and connect with her wireless key.

I got a message this morning it wasn't working so i advised her to restart both her router, the PC and disconnect her old device while this was happening, this fixed her problem and she's now connected.

Yep, my 70+ year old Aunt, who to be fair isn't that bad when it comes to computers and her Photoshop skills are superb (she has a hobby of restoring old family photos) managed to pretty much set up her own wireless connection having not done it before with very little help once she had the right tools. I'm very proud of her, she only need a little bit of help as she'd carried out the wireless adaptor and network setup flawlessly.

Ive had the same from two bosses as two different clients this week and its driven me mad.

Case 1. The phone system that went wrong

My company attempted to deploy a new phone system for a client with a very angry boss. he also believes himself to be a technical guru, when in fact he is not.

I get called into these things as I understand networks, I'm nothing to do with Phone system themselves.

Long story short, the phone suppliers provisioning server had some incorrect information that was causing half the phones to home to a phone server address from years ago. However at 10pm at night the people deploying this didn't know it, and so the boss of the client is searching for any reason for this to be happening

and he thinks he has found it. No, he doesnt think, HE KNOWS!!!

He's on the meraki config. he's checking the difference between the port config of a phone that is working and one that has come up with the wrong config.

He's found a difference

RSTP is set to forwarding on the port on the working phone, and is disabled on the port of the non working phone.

The boss is in angry mode, which means, if he doesnt understand something he gets even angrier. It took 45 minutes to explain to him that the settings for Rapid Spanning Tree Protocol for those phone ports would have absolutely nothing to do with why phone 1 has pulled the correct config from a provisioning server on the Internet, and why phone 2 has pulled the incorrect config from the same provisioning server on the internet, and to give him a breakdown of what RSTP actually is.

​

Case 2: Brickdust VS ILO

a client had some new cables run, drilling was involved, and then the phonecall with the magic words came in "EVER SINCE YOUR...." which in this case "Ever since your man was in and doing drilling, our server has been making really loud noises like it's going to take off, it must be clogged with brick dust"

I'm typing this right now on the site.

i took the case off, a minor bit of dust clogging, but no brickdust. gave the server a dose of airduster just in case.

then what do we see. ILO Errors. The flash in this 7 year old server is going, and ILO errors that align with the fans going into take off mode.

a flash of firmware and a reset of the flash stoarge seems to have calmed it down, but if it comes back then the answer is to buy a warranty and get the mainboard replaced.

but reality means i am going to have to argue with the client, that despite articles from HP and error logs he's still going to blame it on the drilling.

So, a few days ago it was a fairly normal day in the office. My compadre and I are busier than usual--we're finalizing setup upstairs for a training meeting next week that required us to scrounge up every unused PC that isn't an antique, preparing a package to get a new location opened, and that's all on top of tickets that range from daily database mainenance to helping the hapless to the occasional hairpuller where we have to try and get help from vendors that don't exactly have the best customer service. All in all, we're pretty focused on our workload.

Meanwhile--whack...whack...whack--we hear a manager who's in the office from their store this week hitting his phone to try and get it to work. It's not abnormal to hear percussive maintenance going on from time to time--folks getting frustrated with their mice or keyboards when their PC locks up for a few seconds while processing something, so we mostly brush it off. I wind up taking a long call to resolve some issue at another store.

whack...whack...whack

Still on the call, trying to sort out some connection issue with a printer or scanner if I recall.

whack...whack...whack

The manager is starting to just continuously beat this phone. It's getting to the point where it's hard to focus on any other work.

whack...whack...whack

Finally, I get up and check what's going on. This guy is repeatedly whacking the phone--not on the edge of his desk, but on his CANE, all because the screen won't respond (I wonder why) and he's trying to turn off the flashlight feature. I take the phone, use the buttons on the side to turn the phone off and back on, deactivating the flashlight, and hand it back to him.

The scenario: a small business I work with remotely had a power outage that caused a hardware failure in a local server (handles the point of sale (POS) software and shared files). This meant the client computers couldn't connect to the POS, but they still had email/internet because the server doesn't handle that.

I don't think the staff understand what the server does and/or that it is located at their business. I also believe they think the shortcut on their desktop IS the POS.

First I get an email asking if deleting a shortcut will delete the POS. I try to remote into the server and it is offline. So I call them, explaining how to check if the server computer is powered on. I explain how to manually power cycle it. After a few calls/emails back and forth, I say that I think the computer is broken and needs to be repaired. I then get an email that says "we can access the internet and email on our workstations but still can't connect to the POS".

So a few more emails later and I think (I hope) they understand that the POS software is on the server as well as all their shared files, when I get an email saying they're taking the server into a local repair shop (at my request since I am remotely helping them at this point). They then ask me which computer they are taking in (the small router or the mini-atx case computer) - I had a good laugh at that one.

An hour goes by and I get an email: "we rebooted our computer and still cannot connect to the POS". So I ask if the repair shop fixed the server and they powered it back on. "No, that computer is still at the shop being fixed". /facepalm UGHH.

I have found a new respect for those who have to deal with this daily.

This site as the title suggests uses a fax line cable directly to an Ethernet port for sending and receiving faxes on their printer. They also don't have a dedicated RJ11 port for faxing. They do and use a fax server.

This has been an issue for 2 years. I was forced to replace the fax card twice with a new cable each time. They even got a free printer replacement out of this. I let them know after installing the new printer that this will not fix the issue.

Got a nother service request for bad quality faxes and fax issues of not sending or receiving faxes intermittently.

The onsite techs refuse to a RJ11 to RJ45 adapter. I would prefer they just unplug the line from the fax modem and use the Ethernet cable and port that is already being used by the printer. Then, set the printer up on the already existing fax server.

The company I work for is old.

My supervisor has worked at this company for 40 years. (I hear upward momentum will be possible in the next 5 years or so)

There are 2 classes of employees, the old guard who will never ever be fired for any reason.

And the newbies (10 years or less) who can get dismissed without an act of congress.

At the beginning of this rant and during the events therein, the HelpDesk was averaging 50-90 tickets a day across 4 people, and only maybe 15 of those were password resets and AD unlock requests.

The rest of the problems came in waves with entire departments being knocked out of commission seemingly at random and the fixes sometimes needing 2-3 hours of repair.

Most of these issues can be traced to a single person.

Joe Schmo was a security engineer. He had been with the company for 33 years and has his fingers in every pie and system regardless of if he was supposed to, yet I had to spend an hour getting around his own lack of permissions to install AD on his new laptop.

Joe Schmo has caused the company a lot of headaches, if he was a character from Good Omens he would be Witchfinder Pulsifer.

His last act as an engineer was pushing forth a Prod change that was only to affect 15 Linux users as a test. So, of course it instead made it so that whenever anyone opened a new broswer tab on the Microsoft computers, their computer would crash and burn, some to an unrecoverable state.

When the investigations were tracked back, it was found that the Help Desk had missed a step in the primary troubleshooting and because it was Joe Schmo's program, it turned a false positive.

So the blame was shifted to us, rather than the prod change that Joe put out.

A few months later Joe Died in his desk at home.

It was sad for those that knew him for their entire careers.

But Now it's been over a year and the HD queue has gone over 40 tickets a day only twice since.

More recently.

The ancient hardware that allows the company to keep going had an awful fault that sent off alarms at 3am.

My boss, my Team lead, everyone on Slav- Salary was called in to mitigate the disaster.

As i said before, this company is old and the equipment there in is even older.

The backups we had waiting for this kind of thing also failed because they'd been sitting in a closet for 10 years.

The permafix needed specialist (and expensive) custom machining to get parts fit to spec again.

Somewhere around 4am someone found the right combination of spit, Duct Tape, and paperclips to get the system stable enough to allow the company to operate.

When doing an inspection to make sure nothing else was giving out signs of impending doom, they found one of the old guard in a lonely corner of a sub basement. He had worked on this hardware for 50 years and potentially was part of the original install crew. He was cold and dead, The time of death found to be when the machines started back up again.

The hardware ran on that shoestring fix for 2 days until the speciality parts arrived with backups and maintenance procedures in tow.

Im pretty sure he gave his life to the company in more ways than one, and those old processors took the ultimate price to keep running and keeping our pay checks moving.

I now feel even more justified in telling people that their systems broke or fixed themselves because of ghosts meddling with them.

I am not official IT, I just know enough about our companies computers to keep them running and install new parts when one breaks.

This is a story of one breaking.

Tropical storm Lee is hitting our small Canadian province directly but the hype was mostly nothing for our part of the province. However we happened to have these small little power flickers that nobody would overly care about right? And obviously in a privately owned coffee gas station would have decent battery backups right?

Wrong, I have been maintaining these machines that for the most part work and I was granted funds to buy a new battery backup for one that died. I did not know what happened to this retired machine until today. Turns out the coffee shops main server pc tower had a battery backup that died and needed a replacement, so logically they would grab another battery backup. And what did I find, the dead battery backup that had been replaced powering the server (horribly).

So whenever a power flicker happened the machine would instantly die, that means any coffee order in drive thru and walk in now has to wait 5 minutes for the machine to boot. Multiply that by the 30 minutes I spent scrambling for a solution and you got a hot mess in a coffee shop.

At the 30 minute mark I clued in why did the server die but not the drive thru and I started digging into the forest of wires and see a battery backup happily chugging along. I grab a power bar from our storage and replace the backup power with a power strip and hook the server up to a battery backup and all is good in the world.

TL;DR: cheap company has 2 battery backups die, replaces one broken machine with another, chaos ensues when a storm plays with the power lines

Side note: took apart the machines and found the batteries to be very spicy pillows

So we provide Point of Sale (POS—basically touch screen Cash Registers) systems to a particular clientele.

Over the past month we've had the same person call in repeatedly because their Point of Sale is unable to connect to our server, as a result customers who are depositing money in their online accounts aren't able to use that balance on the POS and transactions happening on the POS are only able to update the out of date balances stored on the local database.

They do their best to get a different agent every time and try and avoid mentioning that we've helped them before—had I filtered the tickets under this account by this person I would have seen that we'd already done this rigamarole like 3-4 times before in the last month.

I went through the whole process of ruling out an issue with our server and then with ruling out an issue with the local POS before determining the holdup was likely their local network, which was outside my ability to troubleshoot. I asked them to have their IT make sure traffic over a specific port was open.

"Oh yes. We know. We've asked them. They keep refusing. They say that could open us to attack."

And like...I guess??? I mean, I'm level 1 IT—the computer equivalent of a janitor or handyman. I just know how to clobber things with my big dumb assortment of troubleshooting steps until it works again. I'm not really the most well-versed in how networks work. But I feel like having month-old out of sync customer data and balances on your cash register is a bit more problematic for your business right this second.

Used to work frontline for a 24/7 call center and remembered my favorite night time call a few weeks ago.

I worked for a financial access company that as part of our services rented out "terminals" which were essentially an HD slapped to a fancy monitor. One night we get a call from a local bartender saying the touchscreen isn't working. Most of these places do not have a mouse and often the issue is just a loose cable or faulty driver. As I am getting remoted in to check the drivers I can hear her loudly telling the bar what a piece of shit company we are yada yada yada. Eventually we get through troubleshooting and despite everything showing properly the bartender keeps saying there's no response on the screen. We decide to put in an after hours call for a replacement PC/Monitor, again this is after hours so we had to wake up a tech to get this done.

​

Tech gets there and does some inital troubleshooting and finds the PC and monitor working perfectly. At this point I don't know what to say and apologize to the tech and ask if he can walk the bartender doing the calibration process in case that was the issue. 5 minutes pass and the same, previously annoyed tech, calls back absolutely dying of laughter as he explains what the issue was.

​

The bartender had very long acrylic nails and was touching the screen with the nails instead of her regular fingers. Once we explained the issue she proceeded to complain we need to get better monitors. When I asked if she does the same thing on her phone she answered "Of course not I don't want to scratch the screen". Have a good day ma'am *click*

At my last job, the L2 IT support team was tasked with assisting with switching our userbase from Skype to Teams. I figured that the transition would go over well enough, barring the normal bumps that come with any software transition. I worked in the public sector in a US State.

However, one supervisor in a customer support decided any change was anathema to their manager style and made it loud and clear to the IT support team. She stated that couldn't monitor her staff remotely without their enforced Skype check-ins.

I often was forced into taking inbound calls due to the L1 team being short-staffed and overstretched as it was. On this day, I was asked to take calls due to the Teams transition.

"Hi, this is L2 with Department"

"Why are you forcing teams onto us??"

"This was a decision by the Admin teams to make sure we have Microsoft support."

"My agents say it's too hard to understand???"

"We have a whole site dedicated too..."

"No no no no! I need someone to sit with each person to explain how it works to them!"

"I don't recall any requests for assistance, most of the call agents work from home full time. I haven't gotten many calls or tickets from the public service desk about it.

"Well, they set themselves to busy and they need to available at all times!!?? They need to taught about how the software works so I can monitor them!!"

"I can point them to the online guide and they can call or submit a ticket if they have an issue or have a question a teammate can't answer"

At this point, I figured that this was a manager having issues adapting to a software that didn't take much training for our userbase to embrace. Skype wasn't doing it for most of the users, including the call agents.

"Well, I don't get it at all. The software is not user friendly!"

"Not user friendly" tends to be mean that a user can't understand an UI within .0002 milliseconds of looking at it.

"It's like some stupid Harry Potter magic and I'm just a person who can't read spellbooks! Explain it to me, oh wizard.

She's clearly seething at this point, eager to take it out on someone after being inconvenienced by a software update.

"Alright, you can tell your team they can call me directly if they have any questions or concerns. I am trained on these unknown magics.

She clearly wants to rant more about how IT decisions need a 12-step approval process. I state that changes are made with everyone's input to and that she should take to (Department head guy with a cool head and is sympathetic to the IT teams). I end the call there.

I end up reaching out to a call agent that was under supervisor lady. I worked before about their Teams install. They were just happy they had something more robust than Skype for work. No issues from them or their team members.

The caller ended up leaving the organization a month later due to "not gelling with the deparment" anymore.

​

(apologies in advance for length, I'm notorious for using 10 words where 1 would suffice)

This happened probably 18 years ago and the company has been out of business for a significant length of time, so it's probably safe to tell this story now.

I once worked for a company who specialised in building retail management software for clothing retailers. One of our customers was run by a very opinionated CEO who thought he knew everything about everything and could be very difficult to deal with if you weren't on his right side.

One day we received a support call from them, stating that their production server had crashed (firmware issue with their new HP servers caused the disks to corrupt themselves) and they needed help with DR. It's worth noting that our system encompassed pretty much their entire operations apart from finance - purchasing, inventory, point-of-sale, telesales, warehousing, everything ran against this one database in an online manner (ie: POS in every store was a web application running online against the one database). Great for real-time data accuracy, terrible in a DR situation because the entire business comes to a halt. Anyway, we went in and asked about their backups. Crickets. The CEO and CTO had been butting heads about DR and didn't have anything finalised, let alone working. So no SQL backups. This is not good.

So we shipped the server and disks off to a forensic data recovery firm and within a couple of days they managed to recover 99% of the database. A support call was logged with HP about the firmware fault, patches installed, the database recovered/rebuilt and brought back online. The company basically ran on paper for those couple of days and had to manually enter 2-3 days of sales/orders data but they essentially got away with the near disaster without serious consequences.

Life goes on, I work on-site with them doing custom development a few days a week and over the next few months I listen in the background as the head-butting between CEO and CIO continue for months about DR strategies. Grand designs about a fully automated fail-over to a second environment are mapped out and costed, then rejected, then new plans drawn up etc etc.

One day around the end of October I'm working on some changes that need up-to-date data I and ask the "DBA" if the Test environment can be refreshed with the backups from the previous night. The answer? No, because the most recent backup that the "DBA" has is 3 months old. Say what?! I literally shout at the "DBA" (yes, now you know why I'm using air quotes for his name) to make a backup of the production database right this second and get some nightlies sorted. Off he goes and makes a backup.

I can't remember now why I didn't end up using that backup for my testing (possibly I just got busy) but a few days later the HP servers suffered the same hiccup as 6 months before and corrupted the disks on the server. Crap. Well, at least this time we have a fairly recent (2-3 days old) backup we can work from, right? Wrong! Yes, there was that manual SQL backup, but this "DBA" had made the backup onto the same disks as the production database. The ones that were now corrupted. And he still hadn't set up any nightly backups to anywhere

This time the forensic guys couldn't work their magic. The database was toast. All we had to go on for getting this system back up and running was the Test database (now 3 months old and with test data that needed cleaning out), spreadsheets, data files, records from their website etc. I spent weeks trying to rebuild customer histories, customer order histories, inventory positions from these files. The warehouse and every store had to perform full stocktakes. There was a 3 month gap in their sales histories that they had to paper over in their financials by resorting to bank transaction. Customers were calling up asking what happened to their website/telesales orders and told "we don't even have you in our system, let alone your order".

The kicker is that they had an electronic Store Credit system, which of course was stored in the database. Return an item? It goes into the system as a Store Credit you can use when you next shop. But with a 3 month gap in sales/returns/order history there was no way of determining a customer's Store Credit balance. So from them on (I don't know exactly how long, I stopped doing work for them soon after) you could walk into any of their stores, go to the counter with some goods and say "oh, I thought I had a $100 Store Credit" and the staff would trust you weren't lying and let you leave without paying a cent.

Kids, never land your only backups on production hardware!

A couple months ago I made this post on TFTS talking about the general incompetence within my team/company, but this week had something happen that made me think I should really post a follow-up.

This has got a little longer than I had intended, but I'm still a little mad, so I added a TL;DR at the bottom
.For those that want the details, lets get on with the story of my manager proving they don't have the first clue about what their team does.

It's not directly a 'tech support' thing, more of general incompetence around a tech upgrade, but as I'm basically only brought into this because I'm the one expected to fix it if it went wrong, here we go.

There's been a new 'super duper all singing all dancing' system that's been spoken about being on it's way 'soon' to help us dealing with customer cases on probably the most important part of our team's work (that I am basically in charge of). My manager had mentioned it a few times, but other than mentioning the broad-strokes improvements we'd never seen any of it - which might seem weird, but that's not super uncommon if it's pretty much a like-for-like upgrade layout/visually, but just much better on the back-end.

For the last couple of months my manager has not been attending meetings (and sending the assistant manager (AM from now on) in their place), barely spoken to the team unless there's a major issue, all because "I'm working on a very important project and it needs a lot of my attention", which we all assume is this project (still no confirmation if it was, but there's almost no other explanation)

This week, up popped a meeting in the calendar with the team that have been working on this system, and since my manager is on annual leave for a holiday abroad, me and the AM were pulled into a meeting on their behalf. Manager had told the AM that it was to run through how it would all work ahead of it going live, but shouldn't be a big deal because they'd had a quick glance over the 'final' version and everything seemed like it would be perfect.

About 45 minutes before the meeting starts, the person from the other team that's running the meeting messages basically saying "You still haven't sent your feedback over ahead of the meeting, can it be sent ASAP so we're not just spending the whole meeting sat around reading over it?", which was obviously not at all what was expected. Turns out we were supposed to have been going through it and raising dummy cases to see how it all works ahead of time. Me and the AM had the new system shared with us, and frantically hopped into a call and raised a few test cases, planning to make notes of anything that should be tweaked.

The system was a complete mess, it was slightly more visually interesting, but still incredibly basic looking. There were no sections for customer information to help us actually identify who tf we're dealing with, and we noticed it asked some very specific questions that could only be there to rule out edge-case problems which, while they do come up from time to time, are not a main priority & there are far more pressing/important issues that we deal with that would be way more useful if we're going to delve further into detail up-front.

After loading two cases, we realised that there was a LOT wrong with this new system, and I sent 14-15 detailed points across, with a paragraph each explaining why things should be changed & potential alternative routes to take. This was was later followed by a further 7 paragraphs after showing the other semi-techy guy on the team the new system & he picked out a few things we'd missed in our rush.

This part may not make much sense since I can't go into detailed specifics, but there was another person on the call who was basically doing the back-end stuff to link this new system into our current system, they said they'd only been told that this was a job that needed doing yesterday, it's not something he's done before, and when they shown us the new system's back-end & it basically looks like it's been cobbled together with paper-mache.

The reply I got from my 20+ paragraphs of feedback was "Everything was all signed-off on as being perfect by stakeholders back in June, so we can't just change something at this point."

We ended up on a call with my manager's manager, who while they knew that the new system was coming in, they'd left my manager in charge of it, as it's related to our department, and when they'd asked my manager how things were going my manager had said everything was running smoothly.

They've ended up having to push the launch back for "at least a few weeks", but I would put money on them having to start the whole design process over again from the ground-up, my manager's manager has basically called out my manager in a chain email that the meeting attendees were CC'd into, saying that they'd been running the project so we needed to discuss things with them about why this has gone so badly, and have a meeting booked in for Tuesday.

Annoyingly, I'm on annual leave for the next 2 weeks so won't be able to get involved, but I'm tempted to dial in from pool-side in Tenerife just to hear wtf is said!

​

TL;DR
- New fantastic system was promised
- Manager has spent months working on it without talking to the people who actually do the job
- Are told that the meeting will just be a final sign-off that we're happy
- It's not & we end up having to fully test it and provide detailed feedback
- It's completely wrong and doesn't cater to what the team needs at all
- Shit is hitting the fan & I'm not going to be around to witness it
- I finish this post and realise that a quick bullet pointed list explains the long rambling post better than the rambling post does.

I used to work IT support for a fairly small company of around 80-100 users. The company was located in Fort McMurray, Alberta and dealt primarily with radio rentals for companies on the Syncrude and Suncor sites. Because of the small volume of tickets, we also did work for contract companies that needed techs to do warranty work for things like Dell, Bell, Lexmark, Xerox, etc.

This one day we get an email with a request from bell to do a line upgrade at a police station in Turner Lake, Saskatchewan. I've never heard of the place and had to look it up. I emailed the contracting company and they insisted they got the right depot, as we were the closest (geographically) to the customer. Turns out we're roughly on the same latitude, and about the same distance from the Alberta/Saskatchewan border on each side of it. Plotting a course to it, however, involved taking an ice road, which wasn't frozen over at this time of year. Otherwise, it was a 12-hour trip one way, down to Edmonton, across the border then north again to Turner Lake.

There was a significant safety issue here as I would have to go alone, and I don't believe there were any accommodations in the small town for an overnight trip. The Safety Officer said no to the drive but decided to arrange a flight for me instead.

He booked a Helicopter Charter from Fort McMurray to Turner Lake. Phoenix Heliflight was the company, with jet black helicopters and orange-red phoenix birds painted on the side. Great company to fly with. I didn't want to think about the bill. Granted, the cost would be charged to the Contractor company, which then would be passed to Bell, but I digress.

So instead of a 12-hour drive, it was a leisurely 45-minute flight by Heli. First time ever on a Helicopter and I will never forget the experience. My manager had to get in on the trip and came along. one sitting in the front with a full 180-degree view, the other in the back and switching places on the way home. The whole trip to another province took merely the morning and we were back in the office.

But the ironic part of the story happens at the customer site.

We ended up landing in their hockey arena parking lot as it's the only place big enough. With snow all over the ground and kicking up high in the air, it's safe to say we attracted quite a bit of attention in the small town. An officer picked us up in his truck and literally drove us to the station, a converted mobile trailer just down the road.

My job, remember, was to upgrade their internet line by flipping a few switched on the back of their router and have Bell direct me on which ones. They would then do a test to make sure it's all good, then I can leave.

So when we arrived, there was a tech already there from SaskTel pulling cables and whatnot, and taking up all the room in the small data closet that held the networking gear. I waited for him to finish while my manager chatted up the officer about the flight. I watched him as he connected cables, hooked up a switch, then he grabbed the modem and started flipping switches on the back.

"Wait, what are you doing?" i asked.

"Oh, I have this work order from Bell to upgrade their internet line." he says showing me his paperwork. I pull out mine and sure enough, its exactly the same job.

He finishes in the closet, calls Bell to test the line, all is good, so he leaves.

I did nothing.

I call Bell. They test the line. All is good.

"Yeah, the upgrade looks good. You can take off now." the rep says.

"Funny you should word it like that..." I laugh. I explained about where I came from and how we took a Heli to get here, only to find a SaskTel Tech had done the job for me. A manager was put on the phone. I repeat the story. He's pissed, but not at me. But not at me as I was just doing my job.

"I'm going to have to talk to dispatch about spending unneeded money..." he says through gritted teeth, "Thanks for your time. Have a safe flight home."

I didn't even open a tool bag. We got aboard the officer's truck, back to the helicopter, then I take the front on the way back.

Like I said, we left like 7am and were back in the office by noon.

So I will begin by saying that as most of us supporting the medical profession know (or find out quick), intelligence in one field does not translate to computers and/or tech in general.

User: C-Suite Higher Up

User2: C-Suite Even Higher Up

Support: ME

​

Email comes in from User 2, stating that they have an error message on their screen that they can't get rid of, no matter what they try. I ask User 2 what the error is stating, and they tell me it's an error regarding an incorrect Zoom link.

Now I'm thinking, "That's weird, should be able to just "x" out of an informational error like that!", so I ask the user for their machine name so I can remote in to assess the situation. When I remote in, I ask the user to show me the error message. Here's where the fun starts.

​

They open their Zoom chat /w USER and show me the SCREENSHOT that USER sent to them /w the Zoom error. User2 states "There it is, and I can't get rid of it at all."

​

I take a deep breath and explain to them the impossibility of closing a static image of the error message.

​

TLDR: Intelligence in one field doesn't always transfer

​

​

​

I'm an application support analyst II for a Silicon Valley company that makes financial software. My job is to handle issue escalations from the leadership teams of two of our biggest platforms. Typically this requires some troubleshooting, and if that doesn't resolve it, I interface directly with the platform developers on the requester's behalf.

Today I received a contact from a manager who was attempting to make outbound calls to a collection of customers that their agents had been working with, but whenever she dialed out, the call would be 'misrouted to a support contact' for another customer she wasn't trying to work with.

I spent a little bit researching, contacted the tier 1s who do a lot more stuff with the softphone we use, determined that they'd never experienced the issue, checked her provisioning and everything looked right, her account was configured correctly. I even tested dialing one of the numbers she was trying and the call connected immediately.

..So I invite the manager to a Zoom session and ask her to share her screen and make an outbound call...

She shares her screen, her softphone is in a post-call status, she immediately goes into available status. As many of you who have worked a call queue before know, going into available status means you'll be receiving a call, and as we're a huge company, one comes in immediately, and there's a guy on the line. She starts stuttering, says "THIS IS AN ACCIDENT" and then immediately HANGS UP on the guy, before turning to me and she's like "SEE, THIS IS WHAT HAPPENS WHEN I TRY AND MAKE AN OUTBOUND CALL?"

I'm pretty sure she could HEAR the facepalm in my voice as I explain to her that she shouldn't be going into available to make outbound calls, and walk her through changing her status into one that takes her out of the queue so she can dial out. She goes red faced and I politely walk her through testing an outbound call (that goes through) while I hold back the "oh my god how did you even get this far?" in my voice, politely encouraging her to use this method to call out going forwards.

The only question I don't want answered is, how many customers had she hung up on between her hire date in 2020 and today?

So I get a ticket from lvl1: User wants to reshuffle her monitors, but there is no option to save it. First off, good job lvl1. Why not file a ticket for yourself as well, since you also do not have that button?

Anyway, I go on the user's PC with remote assist and start shuffling things around, asking her if it is alright each time. She is perplexed, how can I be so clueless about what she wants? I try to explain that she is at home, I am in the office, and as such, I have some difficulty imagining what her setup looks like.

We try all possible combinations of screen arrangements and primary screen settings. Like, literally go through each and every one of them. It is still not the way she wants them. Which is mathematically impossible, unless she wants a nonsense arrangement. "here, this is what its like for my colleague! I want it like this." *attaches a printscreen into the chat.*

Well alright, here. I copied that exact setup. Its still not right. "I thought you would help me." Again, I try to explain that unless she intends to start livestreaming her desk with a camera, I am not exactly sure what else could physically be done to help her. I even advised her to play around with the screen setup window and put them the way she wants them to be. She then got the idea that I am accusing her of playing around. "I thought you were here to help me!?" After a bit further explanation about how grammar works, and exclamations about how hard it is to do the loads of work like this, the ticket got closed with an inconclusive "not solved". Someone else can have fun with that if she insists...

Back in the good old times with bulky monitors.

I used to support a department for 2-3 years, then they moved office. Our contract wasn’t prolonged.

Several months later I had an appointment with another company in the new building.
The new IT guy for the department I knew approached me and asked if I could help him out. Sure, what’s up?

One of the workers complained about her monitor setup since they were in the building. He had tried everything. No resolution was correct. AFAIR she even got a new monitor.

To add more shame, she continually told him “I’m sure fraumausl could solve it instantly”.

Poor guy was desperate.

I entered the office, big hello from the ladies.
It was very hard not to laugh. IT guy was so curious for the solution.

“Mrs X, you now have a new desk. Your monitor is now at the correct distance for typing.
Move your head forward 50cm, does it look like before?”

Mrs X moves forward, brightens up - YES, that’s it. When she’s got her nose at the monitor it’s just like before.
I tell her it’s much more ergonomic now, she’s happy and promises to now sit correctly.
She then smugly tells him “see, I told you she would know the solution”

IT guy could not know that every time I had to install something on her PC I cursed because she didn’t have a proper setup but the keyboard right in front of the monitor. Much too close for working properly.

Hey r/talesfromtechsupport,

So, I had a user come to me today with a "faulty" monitor. It was one of those top-of-the-line ones, only about two months old. She said it wouldn't turn on and claimed she'd tried "everything."

Before diving deep, I did the usual checks: power cables, connectivity, tried a different computer - no luck. The power light wasn’t coming on. As I was contemplating further diagnostics, a strong coffee aroma filled the air. Curious, I tilted the monitor and heard a sloshing sound. You guessed it: coffee inside the monitor.

I asked her about it. She looked genuinely surprised and recalled an "incident" from last week but believed she'd caught the spill in time. Apparently not.

So today’s lesson folks: Monitors aren’t coffee-proof, no matter how high-end they are.

Until the next tech mystery!

I had a call from an old customer of mine, in both senses.

"I can't turn my PC on. I can hear a fan noise, and then nothing. It took me 15 minutes of trying until it came on.

I run through the usual telephone troubleshooting to see if I could save a visit.

To me it sounds like maybe a failing motherboard. I visit with a spare power supply in the car to see if that's the issue.

On site, I shut it down (he'd got it running), and restarted it with the case open.

The CPU fan spins fast and noisily for a couple of seconds, then slows down. Three seconds later it beeps and boots normally.

Clearly he'd heard the fan spin down, assumed it had failed, and kept hitting the power button again after 2.5 seconds just prior to the monitor lighting up. He'd had this particular PC for 13 months and not had this problem before.

I politely suggested it might be an intermittent fault, & that he should let me know if it happens again so I can investigate properly. He agreed happily. I could tell he'd figured out the true problem.