Q I have recently been humiliated by my ISP for spamming. It turned out that a forum I had set up on my home box had been hacked and I was mass mailing the whole world. That issue has been fixed but now I'm losing sleep fearing that my server is a zombie. Do you recommend tools or frameworks for quickly and reliably telling if a machine has been broken into?
A While there is no short answer to that, there are some simple steps that can be followed to reveal most common scripted break-ins. 1. Use ls -lai to determine whether there are any files in /tmp and /var/tmp that shouldn't be there. In particular, watch out for executables, scripts and text files that are full of email addresses. 2. On RPM-based systems it is possible to verify whether system utilities such as Ls, Ps, Netstat and so forth have been replaced with ones that hide the hacker's activity. On a Red Hat-based system, the following packages should be verified using
rpm -V <packagename>
for the following packages: util-linux, coreutils, net-tools, procps and lsof. 3. Check running processes with ps -auxf. 4. Use netstant -tanp to find out whether there are processes listening on strange ports, or inexplicable amounts of outgoing traffic. The -p option shows which program is being used. Very often this is named in a way to make it look like a legitimate program (such as httpd). The lsof command can also list listening ports. 5. Review /etc/passwd to see if any users have been added to the system or have had their UID changed. It's a good idea to compare to a known clean copy of the password file. 6. Check Apache log files for tell-tale signs of exploits where utilities such as Wget were used to download some form of malware. Check other system log files for anything suspicious- in particular, for log files that have been redirected to /dev/null. 7. Finally, Chkrootkit (www.chkrootkit.org) checks your server for signs of rootkit presence.