Well, most of the stuff is up and running (apart from all mailing lists). The Finnish blog awards are now back up and running, and even my normal email works now!
Here's a quick rundown on what happened:
- On Saturday, at about 23:25 person A using a machine from Brazil executed a series of commands using an awstats vulnerability (yes, we had it patched to the latest stable; no, apparently it was not enough).
- He was quiet for about 20 minutes, but at about 23:35 two other attackers B and C (or the same) from Italy and UK almost simultaneously launched a similar attack on the server.
- Person B was able to run "adduser" at 23:45 and add himself an account, logging in and promply downloading a rootkit which allowed him to have root privileges
- Person B then attempted to deface the site, but failed (thanks to the pretty hairy configuration we have over here)
- Person A returned at this point, and tried to execute a new attack, suggesting that he was not able to gain access before
- Person B ran "rm -rf /" on the server, starting to delete everything at about 23:55, presumably to cover his traces. Our logs end at 0:06, when the final daemons failed.
- I received first warning at 0:15. Luckily memory-resident processes kept running for some time, so I was able to inspect the situation and the machine was physically disconnected at about 1 am.
Sunday was mostly used to reinstall a completely new system and do a forensics analysis on the deleted partitions. Sleuthkit turned to be invaluable in reconstructing the deleted local log files (so yes, we have the exact times, methods, and IP addresses). Yes, it works on ext3 as well.
I have backed up most of the necessary stuff daily, so there is little that was lost permanently. Unfortunately I had not stored all the necessary config files, which is why system recovery took longer than expected. Also, due to an oversight none of the mailing lists were backed up, so once we have them established again, ya'll have to resubscribe. Very sorry about that :-/
Back to weblog
|"Main_blogentry_140305_1" last changed on 14-Mar-2005 20:31:24 EET by JanneJalkanen.|