this morning while checking the helpdesk I got a suprise .. 3 different servers for the same client had apache down, a manual restart would not bring it back to live, and worst .. no errors on the log.
upon further investigation I found this on rc.local
/etc/rc.d/init.d/.incsshd -p 31221
unfortunately on of my techs also found it and deleted it .. seems the system was compromised .. but the other two .. no indication of any breaches.
the kernels are 2.4.18’s .. so I know they have the ptrace exploit .. time to do some recompiling