Know your (u)limits

If you run a stress test on your new and shining tomcat box you may notice a bunch of nasty "Out of memory" errors and you may even be forced to restart you Tomcat service. I was.

What to do ?

First, check your memory settings in the /etc/inid.d/tomcat start-up file. Add the well known java memory switches in a line like this one :

JAVA_OPTS="-Xms256m -Xmx512m -Xss256k -server -Dfile.encoding=UTF-8"

Don't forget to change the values according to your case.

But the problem might not go away. Check the system limits for the user that run Tomcat.
Type, logged as the "tomcat" user :

ulimit -a

I was especially interested by the item max user processes because it was different between my remote hosted Linux box (showing out memory errors, with a value of 150) and my local "wmware" Centos (fine, no memory problems, the value was 4096).

On the other hand, in the Tomcat's server.xml, observe this line :

<connector port="80" maxhttpheadersize="8192" maxthreads="1500" minsparethreads="25" maxsparethreads="75" enablelookups="false" redirectport="8443" acceptcount="100" connectiontimeout="20000" disableuploadtimeout="true">

In my case, the value of the maxThreads attribute (1500) was way greater than the ulimit value on the remote box, but well below the value of the local "wmware" server.

So, to make a long story short, I changed the max user processes value on the remote box to a sensible value and all went fine.

There are the steps to accomplish this task :

• edit (as root) the file /etc/security/limits.conf
• add (or modify) the lines

  *   soft   nproc   4096 change this,
  *   hard   nproc   6000 and this value

  Eventually you will want to replace the stars * with the name of the "tomcat" user.
  If you delete (or omit) the lines, the value will be set to unlimited.
• Now, verify the settings by logging-in with the "tomcat" user and type the ulimit -a command and check if the value is the one you want (4096 in our case)
• Back to our server.xml, you can modify, if you want the maxThreads attribute to a greater value (but lesser or equal to the ulimit value).
• Restart Tomcat

And do the stress test again ....

Fight the brute (BFD)

If you are running a public server, you will soon notice that you are the target of brute force attacks. Examine the file /var/log/secure (and secure.1, secure.2 if present) and see for yourself.

We need something automated to fight this. Something that will detect any brute force attempt and make a firewall rule to keep them outside. Now enter BFD (Brute Force Detection).

This page is your friend.

Become root and type :

wget http://www.rfxnetworks.com/downloads/bfd-current.tar.gz
tar -xvzf bfd-current.tar.gz
cd bfd*
./install.sh

Edit now the configuration file /usr/local/bfd/conf.bfd. Change the text

ALERT_USR="0"

to

ALERT_USR="1"

And

EMAIL_USR="root"

to

EMAIL_USR="you@yoursite.com" this is your email, of course

And start bfd with :

/usr/local/sbin/bfd -s

Yap, it's done ;-) But be warned : bfd work together with apf, so be sure to have-it up and running (read the previous post to learn how).

Just one more thing before closing up. After a few days and a few attacks, when you are fine with your bfd setup, it might be wise to ditch maybe 90% of the attacks by changing the default SSH port (22).
You need to do this :

• edit the file /etc/ssh/sshd_config
• find the line

  #Port 22

  and change-it to

  Port 333 replace with your preference

• find the line

  #Protocol 2, 1

  and replace with

  Protocol 2

• DON'T FORGET THE FIREWALL !!
  Add the new ssh port to your firewall settings (/etc/apf/conf.apf) or you will lock yourself out of your own server.
  
• restart ssh

  service sshd restart

Done.

The first firewall

It seems that for Centos the first and simplest option is Advanced Policy Firewall. Let's see if it's enough or we need something fancier.

Install it using the instructions from this page. Execute the following, as root :

cd /usr/src
wget http://rfxnetworks.com/downloads/apf-current.tar.gz
tar -xvzf apf-current.tar.gz
rm -f apf-current.tar.gz
cd apf*
sh ./install.sh
cd /etc/apf

Now edit the file conf.apf and enter the settings you need. The following line is especially useful :

IG_TCP_PORTS="22"

Put here only the ports you need to be accessed Publishfrom outside (only 22 for ssh, 80 for web and 3306 for mysql, in my case).

Warning : be sure to put there the SSH port (22 by default). Otherwise you will lock yourself out of your server.

If you are on a hosting service, edit the file /etc/apf/allow_hosts.rules and add there the addresses of the monitoring servers, if you benefit from such a service.

Now type

service apf start

to start the service and we are ready.

Tomcat on 80

Of course, we don't want to run Tomcat on the default port 8080. We want it on 80.

To change that, we go to /opt/tomcat/conf, edit server.xml and change the line

connector port="8080" ...

to

connector port="80"...

Ready ? Only if you want to run Tomcat as root. But we don't want this, from a security point of view. But, on Linux, only the root can use the "lower" ports.

So we have to dig deeper.
First install gcc if it's not already installed :

yum install gcc

and Autoconf :

yum install autoconf

Now go to /opt/tomcat/bin (you will change this to your path to the Tomcat install dir) and do the following (taken from the Tomcat manual):

tar xvfz jsvc.tar.gz
cd jsvc-src
autoconf
./configure --with-java=/usr/java/jdk1.5.0_11 you will adapt this to your Java install path
make
cp jsvc ..
cd ..

Now you can start Tomcat as a daemon using jsvc. It's an extremely ugly command line. But you can download my tomcat start-up file, un-zip-it, change it to your needs and situation, put it in the /etc/init.d, set the appropriate execute permissions and then run :

chkconfig --add tomcat
service tomcat start (or stop)

And that's it.