Forensic analysis

Well, most of the stuff is up and running (apart from all mailing lists). The Finnish blog awards are now back up and running, and even my normal email works now!

Here's a quick rundown on what happened:

  • On Saturday, at about 23:25 person A using a machine from Brazil executed a series of commands using an awstats vulnerability (yes, we had it patched to the latest stable; no, apparently it was not enough).
  • He was quiet for about 20 minutes, but at about 23:35 two other attackers B and C (or the same) from Italy and UK almost simultaneously launched a similar attack on the server.
  • Person B was able to run "adduser" at 23:45 and add himself an account, logging in and promply downloading a rootkit which allowed him to have root privileges
  • Person B then attempted to deface the site, but failed (thanks to the pretty hairy configuration we have over here)
  • Person A returned at this point, and tried to execute a new attack, suggesting that he was not able to gain access before
  • Person B ran "rm -rf /" on the server, starting to delete everything at about 23:55, presumably to cover his traces. Our logs end at 0:06, when the final daemons failed.
  • I received first warning at 0:15. Luckily memory-resident processes kept running for some time, so I was able to inspect the situation and the machine was physically disconnected at about 1 am.

Sunday was mostly used to reinstall a completely new system and do a forensics analysis on the deleted partitions. Sleuthkit turned to be invaluable in reconstructing the deleted local log files (so yes, we have the exact times, methods, and IP addresses). Yes, it works on ext3 as well.

I have backed up most of the necessary stuff daily, so there is little that was lost permanently. Unfortunately I had not stored all the necessary config files, which is why system recovery took longer than expected. Also, due to an oversight none of the mailing lists were backed up, so once we have them established again, ya'll have to resubscribe. Very sorry about that :-/



Comments

I hope you are not going to use awstats again.

--66.66.33.200, 15-Mar-2005

(not trying to be smartass)

Things I have configured on my ~100 user login server: - only allow ssh from known good ip's (firewall) or known ISP's netblocks/domains (easier with /etc/hosts.allow) - chroot every damn daemon you can: bind, apache, postfix (I couldn't chroot Apache, but you should be able to since you control all sites on your server) - add "AllowGroups ssh" to /etc/ssh/sshd_config, create group ssh and only add trusted users to that group (setting shell to /bin/false isn't enough for non-login users) - only allow zone transfers to your own dns servers - daily apt-get update && apt-get -u upgrade (or similar, in case you dumped Debian) - install mod_security (www.modsecurity.org) and configure it to be as strict as possible - only allow pop3 and imap over SSL (close ports 110 and 143) - install logcheck, and _read_ the reports - daily remote backups of /etc /var /usr /home (+ others you might have) - mount /tmp and /var with nosuid,nodev

And things I haven't done yet: - remote syslogging to a logserver that's not reachable from the logclient (ie. log A to B, close all other ports from A to B) - install file integrity checker (not sure if these are effective) - tripwires (ie. probe low unused port -> get firewalled automatically) - mount filesystems read-only (done this with nfsroot workstations, haven't tried on servers) - configure /etc/security/limits.conf (not sure if this is effective)

--Kim, 15-Mar-2005

Oh, forgot one more thing.

Setup HA-clustering (easy for Kuukkelit, not sure how easy it would be for for the wikis). This sounds stupid, but in case a machine gets compromised, you can just turn if off and no-one will notice.

I have this set up for the most important domains I host, so that I can take down the server for maintenance and not worry about the sites so much (since they won't be down).

--Kim, 15-Mar-2005

Glad to see that your back up and running.

One can only wonder why people would be stupid enough to break into a computer and then just delete everything. As if that would really cover their tracks from a good sysadmin ;).

--ramin, 15-Mar-2005

Miten toi muuten käytännössä toimii tuo kuukkelin javascriptautomaattitäydennys? Täydentääkö se automaattisesti kaikki pinserin blogilistassa olevan blogit vai ainoastaan ne, joita on ehdotettu.

(Mun blogi täydentyy, kaverin ei, voinko siis päätellä, että joku tykkää mun blogista?)

--Hämärä, 16-Mar-2005

Se täydentää vain ja ainoastaan ne blogit, jotka olivat Pinserin listalla joulukuun lopussa. Jos ei sentään ole oppiva :)

--JanneJalkanen, 16-Mar-2005


More info...     Comments?   Back to weblog
"Main_blogentry_140305_1" last changed on 14-Mar-2005 20:31:24 EET by JanneJalkanen.