I'm always weary whenever I see firewalls in deployment architecture. It almost always spell problems. Problems that one will never realize during user acceptance tests. Simply because a user acceptance test environment will never mirror production environment. Hardware firewalls are not cheap.
Luckily, most firewall problems I had seen always boil down to the same thing.
Closing of idle connections.
Most applications would use connection pools, to reuse existing opened connections, for better performance. It can be for database connections, ldap connections, or any other kind.
However, a problem occurs if a connection is allowed to lie idle over a period of time.
Especially if the pool idle connection timeout is longer than the firewall idle connection timeout.
What usually happens is this:
1. Firewall idle connection timeout is set to 30 mins.
2. Database idle connection timeout is set to 60 mins.
3. Database connection timeout is set to 10 mins.
4. Connection is opened from client to server.
5. Connection is left idle for 40 mins, and the firewall closes the idle connection.
6. Client application retrieves the closed idle connection from pool, and sends a database query.
7. Client application blocks for a 10 mins before timing out, because the connection is already closed.
There are three ways to resolve this problem.
1. Set the connection to 'ping' the server while idle from time to time. This could be a select 1 query while idle.
2. Set the database pool idle timeout to be a value lower than the firewall idle connection timeout value (if you are allowed to)
3. This is the more common approach, which is to set the firewall idle connection timeout value to a value higher than the database pool idle timeout. This sounds exactly the same as the 2nd way, but people are usually more inclined to modify firewall settings than application settings.
Posts
