Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.
Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.
Rolled back to the backup before I made it public and now I have a security checklist.
Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.
When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.
questionable?
Yeah just like that. Ask more questions
Technically it’s still a public server. Just even more so.
I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.
I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.
We all make mistakes!
There’s a 40 year I.T. veteran here that still suffers imposter syndrome. It’s a real thing I’ve never been able to shake off
Just look at who is in the White House, mate - and not just the president, but basically you can pick anyone he’s hand-picked for his staff.
Surely that’s an instant cure for any qualified person feeling imposter syndrome in their job.
Although disabling the root user is a good part of security, leaving it enabled should not alone cause you to get compromised. If it did, you were either running a very old version of OpenSSH with a known flaw, or, your chosen root password was very simple.
The latter. It was autogenerated by the VPS hosting service and I didn’t think about it.
It should be a serious red flag that your VPS host is generating root passwords simple enough to get quickly hacked.
I’m pretty sure they assumed if you bought their service, you have the competency to properly set it up.
And I proved them wrong.
One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.
“Hey, why is this Valheim server running like shit?”
“Wtf is
xrx?”“Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”
So anyway, now I use NixOS.
Good point about a default deny approach to users and ssh, so random services don’t add insecure logins.
sudo sudokuYeah, about this; any ssh server that can be run as user and doesn’t do shenanigans like switching user?
I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.
I’m interested, how do you expose your services (on your PC I assume) to the internet through wireguard? Is it theough some VPN?
VPN’s are neat, besides the fact they’re capable of masking your IP & DNS they’re also capable of providing resources to devices outside a network.
A good example is the server at my work is only accessible on my works network, to access the server remotely without exposing it directly to the internet would be to use a VPN tunnel.
Wireguard IS a VPN. He has somehow through his challenges of exposing services to the internet, exposed wireguard from his home to the internet for him to connect to. Then he can connect to his internal services from there.
It’s honestly the best option and how I operate as well. I only have a handful of items exposed and even those flow through a DMZ proxy before hitting their destination servers.
Oh, I thought it was a protocol for virtual networks, that merely VPNs used. The more you know!
Edit: spelled out VPN 😅
I’ve always felt that if you’re exposing an SSH or any kind of management port to the internet, you can avoid a lot of issues with a VPN. I’ve always setup a VPN. It prevents having to open up very much at all and then you can open configured web portal ports and the occasional front end protocol where needed.
Exactly.
All of my services are ‘local’ to the VPN. Nothing happens on the LAN except for DHCP and WireGuard traffic.
Remote access is as simple as pressing the WireGuard button.
This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted
i use su -
This is like browsing /c/selfhosted as everyone portforwards every experimental piece of garbage across their router…
Meh. Each service in its isolated VM and subnet. Plus just generally a good firewall setup. Currently hosting ~10 services plubicly, never had any issue.
Well, if you actually do that, bully for you, that’s how that should be done if you have to expose services.
Everyone else there is probably DMZing their desktop from what I can tell.
Yeah the only thing forwarded past my router is my VPN. Assuming I did my job decently, without a valid private key it should be pretty difficult to compromise.
hey, thats me!
portforwards every experimental piece of garbage across their router…
Man some of those “It’s so E-Z bro” YouTubers are WAY too cavalier about doing this.
Lol you can actually demo a github compromise in real time to an audience.
Make a repo with an API key, publish it, and literally just watch as it takes only a few minutes before a script logs in.
I search commits for “removed env file” to hopefully catch people who don’t know how git works.
–verbose please?
edit: never mind, found it. So there’s dumbasses storing sensitive data (keys!) inside their git folder and unable to configure .gitignore…
yeah, I just tried it there, people actually did it.I always start with .gitignore and adding the .env then making it.
Anywho, there’s git filter-repo which is quite nice and retconned some of my repos for some minor things out of existence :P
My work is transferring to github from svn currently
My condolences
You gremlin lmao
Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.
The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.
Endlessh and fail2ban are great to setup a ssh honeypot. There even is a Prometheus exporter version for some nice stats
Just expose endlessh on your public port 22 and if needed, configure your actual ssh on a different port. But generally: avoid exposing ssh if you don’t actually need it or at least disable root login and disable password authentication completely.
https://github.com/skeeto/endlessh https://github.com/shizunge/endlessh-go https://github.com/itskenny0/fail2ban-endlessh
If it is your single purpose to create a blocklist of suspect IP addresses, I guess this could be a honeypot strategy.
If it’s to secure your own servers, you’re only playing whack-a-mole using this method. For every IP you block, ten more will pop up.
Instead of blacklisting, it’s better to whitelist the IP addresses or ranges that have a legitimate reason to connect to your server, or alternatively use someting like geoip firewall rules to limit the scope of your exposure.
Since I’ve switched to using SSH keys for all auth Ive had no problems I’m aware of. Plus I don’t need to remember a bunch of passwords.
But then I’ve had no training in this area. What do I know
I’ve recently seen login attempts using keys, found it curious…
Probably still looking for hosts that have weak Debian SSH keys that users forgot to replace. https://www.hezmatt.org/~mpalmer/blog/2024/04/09/how-i-tripped-over-the-debian-weak-keys-vuln.html
