Check Linux Server For Rootkits, Check If CentOS Linux Server Is Compromised
Yesterday I wrote an article about securing your /tmp and /var/tmp directories on a Linux server because I had found some files uploaded to the /tmp directory via the apache user. After locking down those directories I wanted to verify that there were no other issues on the server so I installed Rootkit Hunter and Rootcheck which are two applications that will assist you with verifying the integrity of your Linux server. Below is information on installing Rootkit Hunter and Rootcheck as well as information on how to use each of them effectively.
Install Rootkit Hunter On CentOS Linux:
- Download Latest Rootkit Hunter: Open the Rootkit Hunter page on SourceForge to obtain a link to the latest version of Rootkit Hunter or RKHunter by clicking here. The current version is Rootkit Hunter 1.3.6 which can be obtained by clicking on the “install or upgrade to Rootkit Hunter version 1.3.6” link near the top of the page.
- Untar Rootkit Hunter: Now unpack the file using tar with the syntax below which will generate a rkhunter-1.3.6 directory with all of the Rootkit Hunter files inside of it.
tar -zxvf rkhunter-1.3.6.tar.gz
- Install Rootkit Hunter: After unpacking the rkhunter-1.3.6.tar.gz file move into the newly created rkhunter-1.3.6 directory and run the below command to install the Rootkit Hunter.
[root@dev rkhunter-1.3.6]# ./installer.sh --install Checking system for: Rootkit Hunter installer files: found A web file download command: wget found Starting installation: Checking installation directory "/usr/local": it exists and is writable. Checking installation directories: Directory /usr/local/share/doc/rkhunter-1.3.6: creating: OK Directory /usr/local/share/man/man8: exists and is writable. Directory /etc: exists and is writable. Directory /usr/local/bin: exists and is writable. Directory /usr/local/lib: exists and is writable. Directory /var/lib: exists and is writable. Directory /usr/local/lib/rkhunter/scripts: creating: OK Directory /var/lib/rkhunter/db: creating: OK Directory /var/lib/rkhunter/tmp: creating: OK Directory /var/lib/rkhunter/db/i18n: creating: OK Installing check_modules.pl: OK Installing filehashmd5.pl: OK Installing filehashsha1.pl: OK Installing filehashsha.pl: OK Installing stat.pl: OK Installing readlink.sh: OK Installing backdoorports.dat: OK Installing mirrors.dat: OK Installing programs_bad.dat: OK Installing suspscan.dat: OK Installing rkhunter.8: OK Installing ACKNOWLEDGMENTS: OK Installing CHANGELOG: OK Installing FAQ: OK Installing LICENSE: OK Installing README: OK Installing language support files: OK Installing rkhunter: OK Installing rkhunter.conf: OK Installation complete
- Generate Rootkit Hunter File Properties DB: Run rkhunter with the –propupd switch to generate a database of file properties from your server as shown in the below output. This will be used in the future to verify the integrity of files on your server.
[root@dev rkhunter-1.3.6]# rkhunter --propupd [ Rootkit Hunter version 1.3.6 ] File created: searched for 159 files, found 137, missing hashes 4
As you can see above the database was created with 137 files in it and will later be used to verify those files have not been compromised.
Rootkit Hunter has now been installed on your server and is available for use with the rkhunter command which we will discuss in further detail below. First we want to also install the Rootcheck command using the directions below.
Install Rootcheck On CentOS Linux:
- Download Rootcheck: Visit the Trend Micro OSSec Rootcheck page to download the latest version of Rootcheck, which is currently Rootcheck version 2.4, by clicking here. After obtaining the link from the bottom of the page use something like wget to download Rootcheck to your server.
[root@dev src]# wget http://www.ossec.net/rootcheck/files/rootcheck-2.4.tar.gz --2010-09-08 22:46:24-- http://www.ossec.net/rootcheck/files/rootcheck-2.4.tar.gz Resolving www.ossec.net... 18.104.22.168 Connecting to www.ossec.net|22.214.171.124|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 379167 (370K) [application/octet-stream] Saving to: `rootcheck-2.4.tar.gz' 100%[==========================================================================================>] 379,167 808K/s in 0.5s 2010-09-08 22:46:25 (808 KB/s) - `rootcheck-2.4.tar.gz' saved [379167/379167]
- Unpack Rootcheck: Use the tar command to unzip and unpack the contents of the Rootcheck file as shown in the below syntax.
tar -zxvf rootcheck-2.4.tar.gz
- Compile Rootcheck: When unpacking the rootcheck file a new directory will be created called something similar to rootcheck-2.4. Change into the new directory and run the below command to compile Rootcheck.
[root@dev rootcheck-2.4]# make all INFO: Little endian set. Compiling Rootcheck...
The first two lines will be generated during the compile process followed by a couple dozen other lines. Once the process is completed you will be able to run Rootcheck as shown below.
Rootcheck is now compiled and ready to use. In the next section of this article after the Rootkit Hunter example we will show a Rootcheck example.
Rootkit Hunter Example On CentOS Linux:
Below is the example output of Rootkit Hunter on a CentOS Linux Server. Rootkit Hunter will check a bunch of different sections including system commands, rootkits, network, localhost, and application versions. You will be required to hit the Enter key after each section. Make sure you analyze the rkhunter.log file located in /var/log once the process is completed as there will be details about any warnings received during the rkhunter process.
[root@dev ~]# rkhunter --check [ Rootkit Hunter version 1.3.6 ] Checking system commands... Performing 'strings' command checks Checking 'strings' command [ OK ] Performing 'shared libraries' checks Checking for preloading variables [ None found ] Checking for preloaded libraries [ None found ] Checking LD_LIBRARY_PATH variable [ Not found ] Performing file properties checks Checking for prerequisites [ OK ] /bin/awk [ OK ] /bin/basename [ OK ] /bin/bash [ OK ] /bin/cat [ OK ] /bin/chmod [ OK ] /bin/chown [ OK ] /bin/cp [ OK ] /bin/csh [ OK ] /bin/cut [ OK ] /bin/date [ OK ] /bin/df [ OK ] /bin/dmesg [ OK ] /bin/echo [ OK ] /bin/ed [ OK ] /bin/egrep [ OK ] /bin/env [ OK ] /bin/fgrep [ OK ] /bin/grep [ OK ] /bin/kill [ OK ] /bin/logger [ OK ] /bin/login [ OK ] /bin/ls [ OK ] /bin/mail [ OK ] /bin/mktemp [ OK ] /bin/more [ OK ] /bin/mount [ OK ] /bin/mv [ OK ] /bin/netstat [ OK ] /bin/ps [ OK ] /bin/pwd [ OK ] /bin/rpm [ OK ] /bin/sed [ OK ] /bin/sh [ OK ] /bin/sort [ OK ] /bin/su [ OK ] /bin/touch [ OK ] /bin/uname [ OK ] /bin/gawk [ OK ] /bin/tcsh [ OK ] /usr/bin/awk [ OK ] /usr/bin/chattr [ OK ] /usr/bin/curl [ Warning ] /usr/bin/cut [ OK ] /usr/bin/diff [ OK ] /usr/bin/dirname [ OK ] /usr/bin/du [ OK ] /usr/bin/elinks [ Warning ] /usr/bin/env [ OK ] /usr/bin/file [ OK ] /usr/bin/find [ OK ] /usr/bin/GET [ Warning ] /usr/bin/groups [ Warning ] /usr/bin/head [ OK ] /usr/bin/id [ OK ] /usr/bin/kill [ OK ] /usr/bin/killall [ OK ] /usr/bin/last [ OK ] /usr/bin/lastlog [ OK ] /usr/bin/ldd [ Warning ] /usr/bin/less [ OK ] /usr/bin/links [ Warning ] /usr/bin/locate [ OK ] /usr/bin/logger [ OK ] /usr/bin/lsattr [ OK ] /usr/bin/md5sum [ OK ] /usr/bin/newgrp [ OK ] /usr/bin/passwd [ OK ] /usr/bin/perl [ OK ] /usr/bin/pgrep [ OK ] /usr/bin/pstree [ OK ] /usr/bin/readlink [ OK ] /usr/bin/runcon [ OK ] /usr/bin/sha1sum [ OK ] /usr/bin/sha224sum [ OK ] /usr/bin/sha256sum [ OK ] /usr/bin/sha384sum [ OK ] /usr/bin/sha512sum [ OK ] /usr/bin/size [ OK ] /usr/bin/stat [ OK ] /usr/bin/strace [ OK ] /usr/bin/strings [ OK ] /usr/bin/sudo [ OK ] /usr/bin/tail [ OK ] /usr/bin/test [ OK ] /usr/bin/top [ OK ] /usr/bin/tr [ OK ] /usr/bin/uniq [ OK ] /usr/bin/users [ OK ] /usr/bin/vmstat [ OK ] /usr/bin/w [ OK ] /usr/bin/watch [ OK ] /usr/bin/wc [ OK ] /usr/bin/wget [ Warning ] /usr/bin/whatis [ Warning ] /usr/bin/whereis [ OK ] /usr/bin/which [ OK ] /usr/bin/who [ OK ] /usr/bin/whoami [ OK ] /usr/bin/gawk [ OK ] /sbin/chkconfig [ OK ] /sbin/depmod [ OK ] /sbin/fuser [ OK ] /sbin/ifconfig [ OK ] /sbin/ifdown [ Warning ] /sbin/ifup [ Warning ] /sbin/init [ OK ] /sbin/insmod [ OK ] /sbin/ip [ OK ] /sbin/kudzu [ OK ] /sbin/lsmod [ OK ] /sbin/modinfo [ OK ] /sbin/modprobe [ OK ] /sbin/nologin [ OK ] /sbin/rmmod [ OK ] /sbin/runlevel [ OK ] /sbin/sulogin [ OK ] /sbin/sysctl [ OK ] /sbin/syslogd [ OK ] /usr/sbin/adduser [ OK ] /usr/sbin/chroot [ OK ] /usr/sbin/groupadd [ OK ] /usr/sbin/groupdel [ OK ] /usr/sbin/groupmod [ OK ] /usr/sbin/grpck [ OK ] /usr/sbin/kudzu [ OK ] /usr/sbin/lsof [ OK ] /usr/sbin/prelink [ OK ] /usr/sbin/pwck [ OK ] /usr/sbin/sestatus [ OK ] /usr/sbin/tcpd [ OK ] /usr/sbin/useradd [ OK ] /usr/sbin/userdel [ OK ] /usr/sbin/usermod [ OK ] /usr/sbin/vipw [ OK ] /usr/local/bin/perl [ OK ] /usr/local/bin/rkhunter [ OK ] /etc/rkhunter.conf [ OK ] Checking for rootkits... Performing check of known rootkit files and directories 55808 Trojan - Variant A [ Not found ] ADM Worm [ Not found ] AjaKit Rootkit [ Not found ] Adore Rootkit [ Not found ] aPa Kit [ Not found ] Apache Worm [ Not found ] Ambient (ark) Rootkit [ Not found ] Balaur Rootkit [ Not found ] BeastKit Rootkit [ Not found ] beX2 Rootkit [ Not found ] BOBKit Rootkit [ Not found ] cb Rootkit [ Not found ] CiNIK Worm (Slapper.B variant) [ Not found ] Danny-Boy's Abuse Kit [ Not found ] Devil RootKit [ Not found ] Dica-Kit Rootkit [ Not found ] Dreams Rootkit [ Not found ] Duarawkz Rootkit [ Not found ] Enye LKM [ Not found ] Flea Linux Rootkit [ Not found ] FreeBSD Rootkit [ Not found ] Fu Rootkit [ Not found ] Fuck`it Rootkit [ Not found ] GasKit Rootkit [ Not found ] Heroin LKM [ Not found ] HjC Kit [ Not found ] ignoKit Rootkit [ Not found ] iLLogiC Rootkit [ Not found ] IntoXonia-NG Rootkit [ Not found ] Irix Rootkit [ Not found ] Kitko Rootkit [ Not found ] Knark Rootkit [ Not found ] ld-linuxv.so Rootkit [ Not found ] Li0n Worm [ Not found ] Lockit / LJK2 Rootkit [ Not found ] Mood-NT Rootkit [ Not found ] MRK Rootkit [ Not found ] Ni0 Rootkit [ Not found ] Ohhara Rootkit [ Not found ] Optic Kit (Tux) Worm [ Not found ] Oz Rootkit [ Not found ] Phalanx Rootkit [ Not found ] Phalanx2 Rootkit [ Not found ] Phalanx2 Rootkit (extended tests) [ Not found ] Portacelo Rootkit [ Not found ] R3dstorm Toolkit [ Not found ] RH-Sharpe's Rootkit [ Not found ] RSHA's Rootkit [ Not found ] Scalper Worm [ Not found ] Sebek LKM [ Not found ] Shutdown Rootkit [ Not found ] SHV4 Rootkit [ Not found ] SHV5 Rootkit [ Not found ] Sin Rootkit [ Not found ] Slapper Worm [ Not found ] Sneakin Rootkit [ Not found ] 'Spanish' Rootkit [ Not found ] Suckit Rootkit [ Not found ] SunOS Rootkit [ Not found ] SunOS / NSDAP Rootkit [ Not found ] Superkit Rootkit [ Not found ] TBD (Telnet BackDoor) [ Not found ] TeLeKiT Rootkit [ Not found ] T0rn Rootkit [ Not found ] trNkit Rootkit [ Not found ] Trojanit Kit [ Not found ] Tuxtendo Rootkit [ Not found ] URK Rootkit [ Not found ] Vampire Rootkit [ Not found ] VcKit Rootkit [ Not found ] Volc Rootkit [ Not found ] Xzibit Rootkit [ Not found ] X-Org SunOS Rootkit [ Not found ] zaRwT.KiT Rootkit [ Not found ] ZK Rootkit [ Not found ] Performing additional rootkit checks Suckit Rookit additional checks [ OK ] Checking for possible rootkit files and directories [ None found ] Checking for possible rootkit strings [ None found ] Performing malware checks Checking running processes for suspicious files [ None found ] Checking for login backdoors [ None found ] Checking for suspicious directories [ None found ] Checking for sniffer log files [ None found ] Checking for Apache backdoor [ Not found ] Performing Linux specific checks Checking loaded kernel modules [ OK ] Checking kernel module names [ OK ] Checking the network... Performing check for backdoor ports Checking for TCP port 1524 [ Not found ] Checking for TCP port 1984 [ Not found ] Checking for UDP port 2001 [ Not found ] Checking for TCP port 2006 [ Not found ] Checking for TCP port 2128 [ Not found ] Checking for TCP port 6666 [ Not found ] Checking for TCP port 6667 [ Not found ] Checking for TCP port 6668 [ Not found ] Checking for TCP port 6669 [ Not found ] Checking for TCP port 7000 [ Not found ] Checking for TCP port 13000 [ Not found ] Checking for TCP port 14856 [ Not found ] Checking for TCP port 25000 [ Not found ] Checking for TCP port 29812 [ Not found ] Checking for TCP port 31337 [ Not found ] Checking for TCP port 33369 [ Not found ] Checking for TCP port 47107 [ Not found ] Checking for TCP port 47018 [ Not found ] Checking for TCP port 60922 [ Not found ] Checking for TCP port 62883 [ Not found ] Checking for TCP port 65535 [ Not found ] Performing checks on the network interfaces Checking for promiscuous interfaces [ None found ] Checking the local host... Performing system boot checks Checking for local host name [ Found ] Checking for system startup files [ Found ] Checking system startup files for malware [ None found ] Performing group and account checks Checking for passwd file [ Found ] Checking for root equivalent (UID 0) accounts [ None found ] Checking for passwordless accounts [ None found ] Checking for passwd file changes [ None found ] Checking for group file changes [ None found ] Checking root account shell history files [ OK ] Performing system configuration file checks Checking for SSH configuration file [ Found ] Checking if SSH root access is allowed [ Warning ] Checking if SSH protocol v1 is allowed [ Not allowed ] Checking for running syslog daemon [ Found ] Checking for syslog configuration file [ Found ] Checking if syslog remote logging is allowed [ Not allowed ] Performing filesystem checks Checking /dev for suspicious file types [ Warning ] Checking for hidden files and directories [ Warning ] Checking application versions... Checking version of GnuPG [ OK ] Checking version of Apache [ Warning ] Checking version of Bind DNS [ Warning ] Checking version of OpenSSL [ Warning ] Checking version of PHP [ OK ] Checking version of Procmail MTA [ OK ] Checking version of OpenSSH [ Warning ] System checks summary ===================== File properties checks... Files checked: 137 Suspect files: 10 Rootkit checks... Rootkits checked : 253 Possible rootkits: 0 Applications checks... Applications checked: 7 Suspect applications: 4 The system checks took: 4 minutes and 15 seconds All results have been written to the log file (/var/log/rkhunter.log) One or more warnings have been found while checking the system. Please check the log file (/var/log/rkhunter.log)
Again make sure to view the rkhunter.log file in detail and work towards correcting any of the issues. After the above run of Rootkit Hunter I ended up upgrading apache, openssl, openssh, and bind on the server above along with investigating in detail some of the other files it provided warnings about. Some of the files needed to have permissions updated and others were OK.
Rootcheck Example On CentOS Linux:
Below is the example output of Rootcheck on a CentOS Linux Server.
[root@dev rootcheck-2.4]# ./ossec-rootcheck ** Starting Rootcheck v0.9 by Daniel B. Cid ** ** http://www.ossec.net/en/about.html#dev-team ** ** http://www.ossec.net/rootcheck/ ** Be patient, it may take a few minutes to complete... [INFO]: Starting rootcheck scan. [OK]: No presence of public rootkits detected. Analyzed 268 files. [OK]: No binaries with any trojan detected. Analyzed 79 files. [OK]: No problem detected on the /dev directory. Analyzed 261 files [OK]: No problem found on the system. Analyzed 37501 files. [OK]: No hidden process by Kernel-level rootkits. /bin/ps is not trojaned. Analyzed 32768 processes. [OK]: No kernel-level rootkit hiding any port. Netstat is acting correctly. Analyzed 131072 ports. [OK]: The following ports are open: 22 (tcp),25 (tcp),111 (tcp),111 (udp), 631 (tcp),631 (udp),889 (udp),892 (udp), 895 (tcp),5353 (udp),41268 (udp),52086 (udp) [OK]: No problem detected on ifconfig/ifs. Analyzed 2 interfaces. - Scan completed in 63 seconds. [INFO]: Ending rootcheck scan. [root@hunter rootcheck-2.4]#
In the above Rootcheck example no issues were located. Even though the Rootcheck output shows no issues you should verify things such as the list of ports that are approved to make sure that you have applications running on all of those ports on your Linux server.