Bogdan Farca's blog

Know your (u)limits

2007-02-26T06:21:00.000-08:00

If you run a stress test on your new and shining tomcat box you may notice a bunch of nasty "Out of memory" errors and you may even be forced to restart you Tomcat service. I was.

What to do ?

First, check your memory settings in the /etc/inid.d/tomcat start-up file. Add the well known java memory switches in a line like this one :

JAVA_OPTS="-Xms256m -Xmx512m -Xss256k -server -Dfile.encoding=UTF-8"

Don't forget to change the values according to your case.

But the problem might not go away. Check the system limits for the user that run Tomcat.
Type, logged as the "tomcat" user :

ulimit -a

I was especially interested by the item max user processes because it was different between my remote hosted Linux box (showing out memory errors, with a value of 150) and my local "wmware" Centos (fine, no memory problems, the value was 4096).

On the other hand, in the Tomcat's server.xml, observe this line :

<connector port="80" maxhttpheadersize="8192" maxthreads="1500" minsparethreads="25" maxsparethreads="75" enablelookups="false" redirectport="8443" acceptcount="100" connectiontimeout="20000" disableuploadtimeout="true">

In my case, the value of the maxThreads attribute (1500) was way greater than the ulimit value on the remote box, but well below the value of the local "wmware" server.

So, to make a long story short, I changed the max user processes value on the remote box to a sensible value and all went fine.

There are the steps to accomplish this task :

edit (as root) the file /etc/security/limits.conf
add (or modify) the lines
```
*   soft   nproc   4096 change this,
*   hard   nproc   6000 and this value
```
Eventually you will want to replace the stars * with the name of the "tomcat" user.
If you delete (or omit) the lines, the value will be set to unlimited.
Now, verify the settings by logging-in with the "tomcat" user and type the ulimit -a command and check if the value is the one you want (4096 in our case)
Back to our server.xml, you can modify, if you want the maxThreads attribute to a greater value (but lesser or equal to the ulimit value).
Restart Tomcat

And do the stress test again ....

Fight the brute (BFD)

2007-02-18T02:56:00.000-08:00

If you are running a public server, you will soon notice that you are the target of brute force attacks. Examine the file /var/log/secure (and secure.1, secure.2 if present) and see for yourself.

We need something automated to fight this. Something that will detect any brute force attempt and make a firewall rule to keep them outside. Now enter BFD (Brute Force Detection).

This page is your friend.

Become root and type :

wget http://www.rfxnetworks.com/downloads/bfd-current.tar.gz
tar -xvzf bfd-current.tar.gz
cd bfd*
./install.sh

Edit now the configuration file /usr/local/bfd/conf.bfd. Change the text

ALERT_USR="0"

ALERT_USR="1"

And

EMAIL_USR="root"

EMAIL_USR="you@yoursite.com" this is your email, of course

And start bfd with :

/usr/local/sbin/bfd -s

Yap, it's done ;-) But be warned : bfd work together with apf, so be sure to have-it up and running (read the previous post to learn how).

Just one more thing before closing up. After a few days and a few attacks, when you are fine with your bfd setup, it might be wise to ditch maybe 90% of the attacks by changing the default SSH port (22).
You need to do this :

edit the file /etc/ssh/sshd_config
find the line
#Port 22
and change-it to
Port 333 replace with your preference
find the line
#Protocol 2, 1
and replace with
Protocol 2
DON'T FORGET THE FIREWALL !!
Add the new ssh port to your firewall settings (/etc/apf/conf.apf) or you will lock yourself out of your own server.
restart ssh
service sshd restart

Done.

The first firewall

2007-02-18T02:38:00.000-08:00

It seems that for Centos the first and simplest option is Advanced Policy Firewall. Let's see if it's enough or we need something fancier.

Install it using the instructions from this page. Execute the following, as root :

cd /usr/src
wget http://rfxnetworks.com/downloads/apf-current.tar.gz
tar -xvzf apf-current.tar.gz
rm -f apf-current.tar.gz
cd apf*
sh ./install.sh
cd /etc/apf

Now edit the file conf.apf and enter the settings you need. The following line is especially useful :

IG_TCP_PORTS="22"

Put here only the ports you need to be accessed Publishfrom outside (only 22 for ssh, 80 for web and 3306 for mysql, in my case).

Warning : be sure to put there the SSH port (22 by default). Otherwise you will lock yourself out of your server.

If you are on a hosting service, edit the file /etc/apf/allow_hosts.rules and add there the addresses of the monitoring servers, if you benefit from such a service.

Now type

service apf start

to start the service and we are ready.

Tomcat on 80

2007-02-18T01:53:00.000-08:00

Of course, we don't want to run Tomcat on the default port 8080. We want it on 80.

To change that, we go to /opt/tomcat/conf, edit server.xml and change the line

connector port="8080" ...

connector port="80"...

Ready ? Only if you want to run Tomcat as root. But we don't want this, from a security point of view. But, on Linux, only the root can use the "lower" ports.

So we have to dig deeper.
First install gcc if it's not already installed :

yum install gcc

and Autoconf :

yum install autoconf

Now go to /opt/tomcat/bin (you will change this to your path to the Tomcat install dir) and do the following (taken from the Tomcat manual):

tar xvfz jsvc.tar.gz
cd jsvc-src
autoconf
./configure --with-java=/usr/java/jdk1.5.0_11 you will adapt this to your Java install path
make
cp jsvc ..
cd ..

Now you can start Tomcat as a daemon using jsvc. It's an extremely ugly command line. But you can download my tomcat start-up file, un-zip-it, change it to your needs and situation, put it in the /etc/init.d, set the appropriate execute permissions and then run :

chkconfig --add tomcat
service tomcat start (or stop)

And that's it.

Some security

2007-01-31T08:54:00.000-08:00

It's time to begin hardening our machine (just a little bit for now).
With the help of this page, we proceed by preventing the root to log remotely to our box. We will provide another user that will become superuser if needed.

First move, create an user account

useradd your-user -G users,wheel
passwd your-passwd

Allow this user to become root, without supplying a password (to be more accurate, all the users of group whell can do that)

visudo

This is a vi editor window so type a to enter edit mode. Uncomment the line

%wheel ALL=(ALL) NOPASSWD: ALL

And then type esc :wq to save and exit.

We have to prevent root to login remotelly : edit /etc/ssh/sshd_config and add or uncomment the line

PermitRootLogin no

We finish all this by restarting the sshd service

service sshd restart

Now, the user we created can log-in with ssh and become super-user with

sudo su -

Done.

Actually I don't like very much to have another user log as root without a password. So I modified back the visudo line to

%wheel ALL=(ALL) ALL

Now the ugly-one : MySql

2007-01-31T06:27:00.000-08:00

Let's start by installing one MySql dependency :

yum install perl-DBI

And now go to MySql site and copy the url of the file to download, from "Red Hat Enterprise Linux 4 RPM (x86) downloads" in our case. We need two files :

one for the server : MySQL-server-standard-5.0.27-0.rhel3.i386.rpm
and another for the client : MySQL-client-standard-5.0.27-0.rhel3.i386.rpm

Let's do some wget (in the /opt folder, just to be consistent) :

wget http://mysql.org/get/Downloads/MySQL-5.0/MySQL-server-standard-5.0.27-0.rhel3.i386.rpm/from/http://mysql.mirror.redwire.net/

wget http://mysql.org/get/Downloads/MySQL-5.0/MySQL-client-standard-5.0.27-0.rhel3.i386.rpm/from/http://mysql.mirror.redwire.net/

Just don't rush to install anything. It won't work (the server, at least). It's because of the dreaded SELinux. Don't ask my what is it. Ask Google ;-) But it prevents some services to run as expected.
We need to circumvent this. It is not easy, but thanks to this post, it's doable :

install SELinux policy sources
yum install selinux-policy-targeted-sources.noarch
now type
setenforce 0
go to /etc/selinux/targeted/src/policy/domains/program (added by the previous install) and edit the file mysqld.te

find the lines (!! all 3)

# because Fedora has the sock_file in the database directory
file_type_auto_trans(mysqld_t, mysqld_db_t, mysqld_var_run_t, sock_file)
')

just below add the following

#
allow mysqld_t var_lib_t:dir { write add_name remove_name };
allow mysqld_t var_lib_t:file { append create lock read write getattr unlink };
allow mysqld_t var_lib_t:sock_file { create getattr unlink };
#

Go back to /etc/selinux/targeted/src/policy and run :
make load
setenforce 1
Ready ! I mean you are ready to continue with the MySql install

Now go back to /opt. And do :

rpm -ivh MySQL-*

And done with the tricky part !

Now let's set a password for the root mysql user :

mysqladmin -u root password 'root'

Testing a little bit :

mysql -proot

You can perform an \s to see the server status. You exit from there with \q

Are we ready ? Nooo .... We must access our server from outside (from our Windows machine, for example, to work on the databases).

Out of the box mysql doesn't allow this. But we can type :

mysql -u root -proot

and then

mysql> grant all privileges on *.* to your-user@'%' identified by your-password with grant option;

Now you can carelessly fire you MySql Administrator or Query Browser and connect using the linux box's IP and the user you just set (your-user with the password your-password).

Tomcat, at last ...

2007-01-29T04:52:00.000-08:00

And now the great moment : enter the big one, Tomcat. With some help from here we will proceed.

To start from somewhere, go to the Tomcat website, copy the url for your prefered version (5.5 at the time of writing) and do the same old wget trick :

wget http://apache.iasi.roedu.net/tomcat/tomcat-5/v5.5.20/bin/apache-tomcat-5.5.20.tar.gz

Be sure to be in the /opt folder, and of course you will replace the example I gave here with your own URL.

Untar :

tar -zxvpf apache-tomcat-5.5.20.tar.gz

and set a symbolic link to the newly created folder

ln -s apache-tomcat-5.5.20 tomcat

Ok for now. Go to /etc/init.d/ and create the file named tomcat. In fact, just copy one already there, for example yum. This will preserve all the file attributes we need.

Edit the file (F4 in MC) and enter the following content :

#! /bin/sh
#
# tomcat Starts tomcat
#
# chkconfig: 2345 98 02
# description: tomcat is a J2EE web application container.
#
export TOMCAT_HOME=/opt/tomcat
export JAVA_HOME=/usr/java/jdk1.5.0_11
[ -f ${TOMCAT_HOME}/bin/startup.sh ] exit 0
[ -f ${TOMCAT_HOME}/bin/shutdown.sh ] exit 0

set -e

case "$1"
in
start)
echo -n "Starting tomcat... "
$TOMCAT_HOME/bin/startup.sh >> /var/log/tomcat 2>&1
echo "started."
;;
stop)
echo -n "Stopping tomcat... "
$TOMCAT_HOME/bin/shutdown.sh >>/var/log/tomcat 2>&1
sleep 1
rm -f $TOMCAT_HOME/logs/*
echo "stopped."
;;
restartforce-reload)
echo -n "Restarting
tomcat... "
$TOMCAT_HOME/bin/shutdown.sh >>/var/log/tomcat 2>&1
sleep 1
$TOMCAT_HOME/bin/startup.sh >>/var/log/tomcat 2>&1
echo "restarted."
;;
*)
N=/etc/init.d/tomcat
echo "Usage: $N
{startstoprestart}" >&2
exit 1
;;
esac

exit 0

Now add tomcat as a service

chkconfig --add tomcat

Start tomcat

service tomcat start

With a little luck you will see

Starting tomcat... started.

Let's test the beast for real :

elinks http://127.0.0.1:8080/

Huraaa !!! It works ! Let's party ...

But wait, don't forget to set the tomcat users. Edit /opt/tomcat/conf/tomcat-users.xml and replace the content with :

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
   <role rolename="manager">
   <role rolename="admin">
   <user roles="admin,manager" password="mypwd" username="myuser">
</tomcat-users>

Of course, you will put the username and password you want. This user is the one that have the ability to manage the server trough the WEB admin console. Don't forget to restart Tomcat :

service tomcat restart

Prepare for Tomcat

2007-01-25T04:05:00.000-08:00

The next big thing is to install my application server, aka Tomcat. But until then, we need some preparations.

First, install elinks. It's a text based browser, running in the console and really neat for testing local pages and downloading packages from the Internet. Again, simple :

yum install elinks

You already know to answer 'y' when prompted. Test the install with :

elinks http://www.sun .com

Type esc for menu and q to exit.

Now, the hard part : install Java, with some help from there. You begin by downloading the JDK kit from the Sun website.

Once there (with the browser from your regular Windows machine for simplicity, or with elinks from the linux console, if you prefer), accept the license agreement, then copy to the clipboard the URL corresponding to the jdk-1_5_0_11-linux-i586-rpm.bin file (right click on the link, copy shortcut). Of course, at the time of reading this, you will download the most recent JDK version.

Fire mc, and go to /opt. Press Ctrl-O to hide the panels.

Now, we do some wget-ing :

wget "the_url_you_copied" -O jdk-1_5_0_11-linux-i586-rpm.bin

Now you have a local copy of the compressed rpm in your /opt folder. You now type

sh jdk-1_5_0_11-linux-i586-rpm.bin

You read the eula with space, you answer yes to the question and you have the thing installed.
Go to /usr/java/jdk1.5.0_11 (you replace this with the jdk version number you installed) and type

./java -version

If by some accident the version returned is not the one you expected, it's probable to hava the "free" version of Java (GCJ) preinstalled. Remove-it :

rm /usr/bin/java

For some unknown reason (unknown for me, of course) the JAVA_HOME environment variable is not set, so go to /etc/profile, edit the thing and add the following lines :

JAVA_HOME="/usr/java/jdk1.5.0_11"
export JAVA_HOME

The environment variable will be set at the next reboot. If you can't wait until then, type

source /etc/profile

Start fighting with the machine

2007-01-18T03:14:00.000-08:00

After the extremely successful attempt in my previous post, the first problems arise.
I deactivated the network on my working laptop (the one that host the vmware player and the centos "test" server) ... After that, the linux box lost his network connection, and refuse to regain-it again.

I started digging for a method to tell linux to use DHCP to fill the eth0 settings. Here it is (learned from here) :

tell etho to go down (if your network interface is not eth0, use ifconfig to identify it):
ifdown eth0
edit (or create) /etc/sysconfig/network-scripts/ifcfg-eth0 and be sure it contains only :
device=eth0
bootproto="dhcp"
onboot="yes"
bring eth0 back up, and all is done :
ifup eth0

My network is again up and running. Feeew !

But, cd-ing and vi-ing trough the linux file system, I starting to wonder if I could use something easier, like Midnight Commander (MC). OK, I admit, I was sure the answer is "Yes", but how ?

It's simple, type:

yum install mc

Answer 'y' when prompted, and you're ready. Really. Type mc and see for yourself.

In the world of CentOs

2007-01-15T02:47:00.000-08:00

In the next postings I will highlight a few findings (nothing original, be assured) in configuring a CentOS box. Why CentOs ? My (future) hosting provider has a very sensible offer for a dedicated server and the cheapest option is a CentOs based box.

So, as I don't think I am some kind of Linux guru, I started by questioning myself about the best (I mean easiest) course of action. Obviously, I need something local to explore and learn, before messing with the production server.

As I hate digging for a long time on new things without seeing anything working, I decided to take the vmware approach, aka downloading a pre-configured centos virtual machine and working with it on my development machine.

I found one here. Thank you c_g-hills !
Of course, before you can do anything with it you must download and install the free vmware player (from here), if it doesn't happen to own the full vmware workstation.

The things are straightforward, and you find yourself running a CentOs server in no-time.

But, you need to configure the pretty boy, install tomcat and mysql and so on.

The first thing I made was to partially deactivate the firewall, in order to access the box trough SSH (with putty). At a later time I will reactivate all the security I need.
For this, you need to enter the following at the linux command prompt :

system-config-securitylevel-tui

In the (pseudo) gui that appear, click the "Customize" button, then check

Trusted Devices: [*] eth0"

Then two OKs, and you're done, for now.

Fire-up putty, enter the server address (ifconfig in the server console if you don't know the server IP) and port (23), and voila, you have remote access to your linux box.

By the way, to cleanly end your putty session, type

exit

But enough for this post ...

First post ...

2006-12-23T01:24:00.000-08:00

First post in my first blog ;-)

Some toughs : this blog is meant to host some of my findings in the eternal quest of writing WEB apps. And a few "fat client" tales, of course.

And please don't be too picky with my English. It's not, by far, my mother tongue.