Tech employee working 50-hour-weeks makes massive mistake impacting 500 servers

I know the technology sector loves overworking employees, for reasons I don't fully understand. Like, it's very common in the video game industry to have crunch times: as the game creeps closer and closer to its release date, developers are forced to work almost around the clock (think 90 hour weeks!) to get it done.

Why can't they just hire more workers? No idea. But it's quite a common thing in this industry. And if workers don't comply with these rules, then they'll be fired with haste.

This dude doesn't work in video games. They work in network security. That is a much more serious position to be in, because they handle not just customer passwords, but like, all of their client's information, spanning 500 servers.

Cheezburger Image 10626309888When you work in network security, the security part is kind of a big deal! So this person was upset to discover that they'd made a huge mistake, and they explained what happened, adding that they're seeking advice.

They use a lot of technical jargon in their story, so let's just read through it and parse what happened with this whole debacle.

pushed unified vuln dashboard with live criticals to public github repo. team is melting down

cannot even process what just happened. we have been grinding for weeks to unify vulnerability data from 12 different security tools into one dashboard.

Two tech workers have a chat in the server room.They then name a bunch of programs that probably only the techiest of techs among us have heard of: "tenable, qualys, snyk, wiz." Okay. Sounds made up, but I'm sure they know what they're doing.

Apis pulling scans, risk scores, everything normalized into single panes so management stops yelling about tool sprawl. Finally got a demo view working Friday. Pulled all the feeds, built the unified queries, even added some fancy risk prioritization graphs. Excited as h*ll so i made a repo to share with the team over weekend.

They seemingly did what the boss wanted, and created a demo version with all the data input.

But then… disaster struck, just as the weekend began. While this person had their 48 hours of free time, private information became public information. Oh. No.

Tech workers holding a serious meeting to discuss how to handle the situation.Forgot to init as private. Pushed to my work GitHub account which is public by default because I use it for side scripts. Commit message was literally 'unified vuln view with prod feeds live check this out team'.

So what they're saying here is that they used an account that is their personal work account, but they left it public for their own ease of access. And apparently, the message when you open this thing was that it was for the team to check out the live results.

Monday morning, Slack explodes. external vuln scanner picks up our repo, indexes it, and now our entire high med crit list from prod environment is scraped and showing in public searches.

They explain that at the crack of Monday morning, their Slack messages are filling up because a vulnerability scanner discovered their mistake. The scanner indexed all of the sensitive information on there, which, as they explain, includes:

customer names, asset tags, cvss scores for unpatched stuff across 500 servers.

The tattletale vulnerability scanner told on them!

One of our biggest clients' assets right there with 'immediate exploit' tags. Heart stopped when I saw it trending in some threat intel feed.

Boss questions team about what to do to secure their servers.They then explain that in their panicked state, they tried to take action. But it was already a bit too late (like, a whole weekend too late, unfortunately). Even after they deleted it, they realized that other people had already scraped the information, as had Google itself. The internet is forever.

Team lead is furious, ciso [Chief Information Security Officer] looping in legal, clients getting calls.

They add that they spent an entire morning doing all that they could to pull back the information, which essentially ruined the whole dashboard they worked on. They're now in a state of blaming themself for the whole thing, wondering how they could've made such a mistake. But perhaps their employer is to blame, because they've been working 50 hour weeks.

If you're working 10 hour days, you're bound to be pushing yourself to the brink. You have less time to sleep, less time to work out and eat healthy meals, less time to let your brain unravel as you watch TV with your partner or read a good book. And a lot of employers have their workers operating this way all the time.

Employee working long hours sleeps at his desk.Here's what's going on now, as the company manages the immense fallout:

Still recovering data feeds without breaking prod scans again. anyone been through this kind of exposure. how bad is the fallout usually. clients gonna bail. need advice on disclosure or cleaning this up before it hits news. please tell me someone has a worse story or fix.

Well, the post made its way to X, and it has numerous comments on Reddit, so safe to say it has breached containment. And with a story this big, affecting 500 servers, I mean… maybe this person should start looking for new work before the news of it all gets too big? Again, not to blame them, but they may want to just look forward instead of sitting around in this giant mess they made.

Workers brainstorm solutions to their tech problemsA lot of people were shocked by the story…

I hope this isn't real, for your sake. If it is, the last thing I would do is advertise it on Reddit and invite more people to come and find it.

You need to take a break from the internet and read a book about operational security

Might want to publicly publish your resume next

This person can't just wait around at their job and be known as "the guy who let 500 servers' worth of client information leak out." There's probably going to be new rules created because of this guy. It's not looking good!

Best of luck to this person, and let's all hope our data wasn't leaked out due to this breach.