Know your (u)limits

If you run a stress test on your new and shining tomcat box you may notice a bunch of nasty "Out of memory" errors and you may even be forced to restart you Tomcat service. I was.

What to do ?

First, check your memory settings in the /etc/inid.d/tomcat start-up file. Add the well known java memory switches in a line like this one :

JAVA_OPTS="-Xms256m -Xmx512m -Xss256k -server -Dfile.encoding=UTF-8"

Don't forget to change the values according to your case.

But the problem might not go away. Check the system limits for the user that run Tomcat.
Type, logged as the "tomcat" user :

ulimit -a

I was especially interested by the item max user processes because it was different between my remote hosted Linux box (showing out memory errors, with a value of 150) and my local "wmware" Centos (fine, no memory problems, the value was 4096).

On the other hand, in the Tomcat's server.xml, observe this line :

<connector port="80" maxhttpheadersize="8192" maxthreads="1500" minsparethreads="25" maxsparethreads="75" enablelookups="false" redirectport="8443" acceptcount="100" connectiontimeout="20000" disableuploadtimeout="true">

In my case, the value of the maxThreads attribute (1500) was way greater than the ulimit value on the remote box, but well below the value of the local "wmware" server.

So, to make a long story short, I changed the max user processes value on the remote box to a sensible value and all went fine.

There are the steps to accomplish this task :

• edit (as root) the file /etc/security/limits.conf
• add (or modify) the lines

  *   soft   nproc   4096 change this,
  *   hard   nproc   6000 and this value

  Eventually you will want to replace the stars * with the name of the "tomcat" user.
  If you delete (or omit) the lines, the value will be set to unlimited.
• Now, verify the settings by logging-in with the "tomcat" user and type the ulimit -a command and check if the value is the one you want (4096 in our case)
• Back to our server.xml, you can modify, if you want the maxThreads attribute to a greater value (but lesser or equal to the ulimit value).
• Restart Tomcat

And do the stress test again ....

Fight the brute (BFD)

If you are running a public server, you will soon notice that you are the target of brute force attacks. Examine the file /var/log/secure (and secure.1, secure.2 if present) and see for yourself.

We need something automated to fight this. Something that will detect any brute force attempt and make a firewall rule to keep them outside. Now enter BFD (Brute Force Detection).

This page is your friend.

Become root and type :

wget http://www.rfxnetworks.com/downloads/bfd-current.tar.gz
tar -xvzf bfd-current.tar.gz
cd bfd*
./install.sh

Edit now the configuration file /usr/local/bfd/conf.bfd. Change the text

ALERT_USR="0"

to

ALERT_USR="1"

And

EMAIL_USR="root"

to

EMAIL_USR="you@yoursite.com" this is your email, of course

And start bfd with :

/usr/local/sbin/bfd -s

Yap, it's done ;-) But be warned : bfd work together with apf, so be sure to have-it up and running (read the previous post to learn how).

Just one more thing before closing up. After a few days and a few attacks, when you are fine with your bfd setup, it might be wise to ditch maybe 90% of the attacks by changing the default SSH port (22).
You need to do this :

• edit the file /etc/ssh/sshd_config
• find the line

  #Port 22

  and change-it to

  Port 333 replace with your preference

• find the line

  #Protocol 2, 1

  and replace with

  Protocol 2

• DON'T FORGET THE FIREWALL !!
  Add the new ssh port to your firewall settings (/etc/apf/conf.apf) or you will lock yourself out of your own server.
  
• restart ssh

  service sshd restart

Done.

The first firewall

It seems that for Centos the first and simplest option is Advanced Policy Firewall. Let's see if it's enough or we need something fancier.

Install it using the instructions from this page. Execute the following, as root :

cd /usr/src
wget http://rfxnetworks.com/downloads/apf-current.tar.gz
tar -xvzf apf-current.tar.gz
rm -f apf-current.tar.gz
cd apf*
sh ./install.sh
cd /etc/apf

Now edit the file conf.apf and enter the settings you need. The following line is especially useful :

IG_TCP_PORTS="22"

Put here only the ports you need to be accessed Publishfrom outside (only 22 for ssh, 80 for web and 3306 for mysql, in my case).

Warning : be sure to put there the SSH port (22 by default). Otherwise you will lock yourself out of your server.

If you are on a hosting service, edit the file /etc/apf/allow_hosts.rules and add there the addresses of the monitoring servers, if you benefit from such a service.

Now type

service apf start

to start the service and we are ready.

Tomcat on 80

Of course, we don't want to run Tomcat on the default port 8080. We want it on 80.

To change that, we go to /opt/tomcat/conf, edit server.xml and change the line

connector port="8080" ...

to

connector port="80"...

Ready ? Only if you want to run Tomcat as root. But we don't want this, from a security point of view. But, on Linux, only the root can use the "lower" ports.

So we have to dig deeper.
First install gcc if it's not already installed :

yum install gcc

and Autoconf :

yum install autoconf

Now go to /opt/tomcat/bin (you will change this to your path to the Tomcat install dir) and do the following (taken from the Tomcat manual):

tar xvfz jsvc.tar.gz
cd jsvc-src
autoconf
./configure --with-java=/usr/java/jdk1.5.0_11 you will adapt this to your Java install path
make
cp jsvc ..
cd ..

Now you can start Tomcat as a daemon using jsvc. It's an extremely ugly command line. But you can download my tomcat start-up file, un-zip-it, change it to your needs and situation, put it in the /etc/init.d, set the appropriate execute permissions and then run :

chkconfig --add tomcat
service tomcat start (or stop)

And that's it.

Some security

It's time to begin hardening our machine (just a little bit for now).
With the help of this page, we proceed by preventing the root to log remotely to our box. We will provide another user that will become superuser if needed.

First move, create an user account

useradd your-user -G users,wheel
passwd your-passwd

Allow this user to become root, without supplying a password (to be more accurate, all the users of group whell can do that)

visudo

This is a vi editor window so type a to enter edit mode. Uncomment the line

%wheel ALL=(ALL) NOPASSWD: ALL

And then type esc :wq to save and exit.

We have to prevent root to login remotelly : edit /etc/ssh/sshd_config and add or uncomment the line

PermitRootLogin no

We finish all this by restarting the sshd service

service sshd restart

Now, the user we created can log-in with ssh and become super-user with

sudo su -

Done.

Actually I don't like very much to have another user log as root without a password. So I modified back the visudo line to

%wheel ALL=(ALL) ALL

Now the ugly-one : MySql

Let's start by installing one MySql dependency :

yum install perl-DBI

And now go to MySql site and copy the url of the file to download, from "Red Hat Enterprise Linux 4 RPM (x86) downloads" in our case. We need two files :

• one for the server : MySQL-server-standard-5.0.27-0.rhel3.i386.rpm
• and another for the client : MySQL-client-standard-5.0.27-0.rhel3.i386.rpm

Let's do some wget (in the /opt folder, just to be consistent) :

wget http://mysql.org/get/Downloads/MySQL-5.0/MySQL-server-standard-5.0.27-0.rhel3.i386.rpm/from/http://mysql.mirror.redwire.net/

wget http://mysql.org/get/Downloads/MySQL-5.0/MySQL-client-standard-5.0.27-0.rhel3.i386.rpm/from/http://mysql.mirror.redwire.net/

Just don't rush to install anything. It won't work (the server, at least). It's because of the dreaded SELinux. Don't ask my what is it. Ask Google ;-) But it prevents some services to run as expected.
We need to circumvent this. It is not easy, but thanks to this post, it's doable :

• install SELinux policy sources

  yum install selinux-policy-targeted-sources.noarch

• now type

  setenforce 0

  go to /etc/selinux/targeted/src/policy/domains/program (added by the previous install) and edit the file mysqld.te
• find the lines (!! all 3)

  # because Fedora has the sock_file in the database directory
  file_type_auto_trans(mysqld_t, mysqld_db_t, mysqld_var_run_t, sock_file)
  ')

• just below add the following

  #
  allow mysqld_t var_lib_t:dir { write add_name remove_name };
  allow mysqld_t var_lib_t:file { append create lock read write getattr unlink };
  allow mysqld_t var_lib_t:sock_file { create getattr unlink };
  #

• Go back to /etc/selinux/targeted/src/policy and run :

  make load
  setenforce 1

• Ready ! I mean you are ready to continue with the MySql install

Now go back to /opt. And do :

rpm -ivh MySQL-*

And done with the tricky part !

Now let's set a password for the root mysql user :

mysqladmin -u root password 'root'

Testing a little bit :

mysql -proot

You can perform an \s to see the server status. You exit from there with \q

Are we ready ? Nooo .... We must access our server from outside (from our Windows machine, for example, to work on the databases).

Out of the box mysql doesn't allow this. But we can type :

mysql -u root -proot

and then

mysql> grant all privileges on *.* to your-user@'%' identified by your-password with grant option;

Now you can carelessly fire you MySql Administrator or Query Browser and connect using the linux box's IP and the user you just set (your-user with the password your-password).

Tomcat, at last ...

And now the great moment : enter the big one, Tomcat. With some help from here we will proceed.

To start from somewhere, go to the Tomcat website, copy the url for your prefered version (5.5 at the time of writing) and do the same old wget trick :

wget http://apache.iasi.roedu.net/tomcat/tomcat-5/v5.5.20/bin/apache-tomcat-5.5.20.tar.gz

Be sure to be in the /opt folder, and of course you will replace the example I gave here with your own URL.

Untar :

tar -zxvpf apache-tomcat-5.5.20.tar.gz

and set a symbolic link to the newly created folder

ln -s apache-tomcat-5.5.20 tomcat

Ok for now. Go to /etc/init.d/ and create the file named tomcat. In fact, just copy one already there, for example yum. This will preserve all the file attributes we need.

Edit the file (F4 in MC) and enter the following content :

#! /bin/sh
#
# tomcat Starts tomcat
#
# chkconfig: 2345 98 02
# description: tomcat is a J2EE web application container.
#
export TOMCAT_HOME=/opt/tomcat
export JAVA_HOME=/usr/java/jdk1.5.0_11
[ -f ${TOMCAT_HOME}/bin/startup.sh ] exit 0
[ -f ${TOMCAT_HOME}/bin/shutdown.sh ] exit 0

set -e

case "$1"
in
start)
echo -n "Starting tomcat... "
$TOMCAT_HOME/bin/startup.sh >> /var/log/tomcat 2>&1
echo "started."
;;
stop)
echo -n "Stopping tomcat... "
$TOMCAT_HOME/bin/shutdown.sh >>/var/log/tomcat 2>&1
sleep 1
rm -f $TOMCAT_HOME/logs/*
echo "stopped."
;;
restartforce-reload)
echo -n "Restarting
tomcat... "
$TOMCAT_HOME/bin/shutdown.sh >>/var/log/tomcat 2>&1
sleep 1
$TOMCAT_HOME/bin/startup.sh >>/var/log/tomcat 2>&1
echo "restarted."
;;
*)
N=/etc/init.d/tomcat
echo "Usage: $N
{startstoprestart}" >&2
exit 1
;;
esac

exit 0

Now add tomcat as a service

chkconfig --add tomcat

Start tomcat

service tomcat start

With a little luck you will see

Starting tomcat... started.

Let's test the beast for real :

elinks http://127.0.0.1:8080/

Huraaa !!! It works ! Let's party ...

But wait, don't forget to set the tomcat users. Edit /opt/tomcat/conf/tomcat-users.xml and replace the content with :

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
   <role rolename="manager">
   <role rolename="admin">
   <user roles="admin,manager" password="mypwd" username="myuser">
</tomcat-users>

Of course, you will put the username and password you want. This user is the one that have the ability to manage the server trough the WEB admin console. Don't forget to restart Tomcat :

service tomcat restart