NGINX reverse proxy in front of Apache: hardened, high performance dedicated server setup

From DP

Jump to: navigation , search

Contents

About

We setup our previous dedicated server after leaving the Laughing Squid Cloud service. It worked ok for about 6months, but then our amateur setup started falling over. Once or twice a week at first, and then every day.

Something had to be done, and by someone more professional than us. Arhi recommended Alex at CloudWebOps.com. Alex did our complete server setup, migration, and hardening at a very reasonable price.

Here are the main features of the setup:

  • Main dedicated server running Linux, Apache, MySQL, and PHP, with a nginx reverse proxy server running in front of everything
  • Secondary Amazon leased server with Munin server monitoring
  • Multiple daily backups to Jungle Disk encrypted backup service
  • Daily SQL database backups to third backup server

If you need a Linux server setup and hardening, you should check out Alex. He went above and beyond, and helped us every step of the way. He even documented the server install on this wiki page - you can try to DIY, but we think the professional setup Alex did was worth it.

Overview

NGINX reverse proxy in front of Apache-HTOP screenshot

When the traffic is too high, Apache generate many high CPU processes and consume too much memory.This situation cause server to crash. Apache is a reliable HTTP server that still holds more than 66% of the web server market, according to W3Techs, but it was not designed with performance or scalability in mind.

You can speed up your current HTTP server by installing a reverse proxy server in front of it. A reverse proxy fetches resources from one or more servers and returns them to the client as if they originated from the proxy server itself.

We will use web server Nginx, Apache serve all dynamic content and Nginx handle all static files without consuming lots of system resources, combining the benefits of both servers.

Hardware

Dedicated Server EX 4

  • Intel® Core™ i7-2600 Quadcore
  • RAM 16 GB DDR3 RAM
  • Hard disks 2 x 3 TB SATA 6 Gb/s HDD 7200 rpm
  • NIC1 GBit OnBoard connected at 100 MBit
  • Traffic 10TB/monthly

Operating System:

  • Debian-60-squeeze-64-minimal (Hetzner image)

Basic server setup - LAMP

Debian OS - upgrade to latest packages

    # apt-get update 
    # apt-get upgrade 


Packages installation

Apache
    # apt-get install apache2
    # a2enmod rewrite
    # /etc/init.d/apache2 restart


configuration:

    # nano /etc/apache2/sites-enabled/000-default
      (default webroot directory: /var/www/)  


check configuration:

    # apachectl -t  


After enabling, disabling, or modifying any part of your Apache configuration, you will need to reload or restart the Apache configuration again with command:

    # /etc/init.d/apache2 reload or 
    # /etc/init.d/apache2 restart




PHP
    # apt-get install php5 php-pear php5-suhosin php5-mysql


configuration: edit /etc/php5/apache2/php.ini Make sure that the following values are set, and relevant lines are uncommented (comments are lines beginning with a semi-colon (;)):

max_execution_time = 60
memory_limit = 128M
error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR
display_errors = Off
log_errors = On
error_log = /var/log/php5.log
register_globals = Off


To apply PHP configuration changes Apache need to be restarted:

    # /etc/init.d/apache2 restart




MySQL
    # apt-get install mysql-server

During the installation, you will be prompted for a password. Choose something secure and record it for future reference. At this point, MySQL should be ready to configure and run. While you shouldn't need to change the configuration file, note that it is located at /etc/mysql/my.cnf for future reference.




ProFTPD
    # apt-get install proftpd

select: standalone mode


configuration:

    # nano /etc/proftpd/proftpd.conf  


check configuration:

    # proftpd -t  


After modifying any part of your ProFTPD configuration, you will need to restart the ProFTPD service:

    # /etc/init.d/proftpd restart




Postfix
    # apt-get install postfix

select: internet site


configuration:

    # nano /etc/postfix/main.cf  
    # nano /etc/postfix/master.cf 


After modifying any part of your Postfix configuration, you will need to restart service:

    # /etc/init.d/postfix restart




List of all installed packages for new installation

    # dpkg --get-selections > all-installed-software.log
    # dpkg --set-selections < all-installed-software.log 
    # dselect (select i -install) 


References:

Advanced server setup - NGINX

Installation

Using repo for Nginx 1.0.11 last stable. For the main Dotdeb repository add these two lines to: /etc/apt/sources.list file

    # deb http://packages.dotdeb.org stable all
    # deb-src http://packages.dotdeb.org stable all


Then fetch the appropriate GnuPG key

    # wget http://www.dotdeb.org/dotdeb.gpg
    # cat dotdeb.gpg | sudo apt-key add -


    # apt-get update    
    # apt-get install nginx

Configuration

Stop the Nginx server if it was started automatically by the package manager and create a new nginx.conf configuration file – installed in /etc/nginx/ by default – by pasting the following and adjusting the paths to those of your installation:

user www-data; #change to the same user apache runs as
worker_processes 8; #change to the number of your CPUs/Cores
worker_rlimit_nofile 8192;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
  worker_connections 1024;
  use epoll;
  accept_mutex off;
}

http {
  server_names_hash_bucket_size 64;
  include /etc/nginx/mime.types;
  default_type application/octet-stream;
  access_log /var/log/nginx/access.log;
  sendfile on;
  tcp_nopush on;
  keepalive_timeout 65;

  # reverse proxy options
  proxy_redirect off;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  # gzip compression options
  gzip on;
  gzip_http_version 1.0;
  gzip_comp_level 6;
  gzip_min_length 0;
  gzip_buffers 16 8k;
  gzip_proxied any;
  gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
  gzip_disable "MSIE [1-6]\.";
  gzip_vary on;

  # include virtual hosts configuration
  include /etc/nginx/virtual.d/*.conf;
}


Nginx should run as the same user Apache runs, to avoid file permission problems.

Besides the proxy setup this configuration file includes some generic performance tuning, such as use epoll as the event model method, which works effectively on Linux 2.6+ kernels. This works in tandem with the next line, accept_mutex off, to improve performance a bit more. Enabling sendfile allows nginx to use the kernel’s sendfile support to send files to the client regardless of their contents. This can help with large static files, such as images, that have no need for a multiple request/confirmation system to be served. Enabling gzip compression for static files can make a big performance difference. The lines starting with gzip enable compression for common web files, such as .css and .js files, on supported browsers.

Apache reverse proxy forward module(mod_rpaf)

If you check the Apache access log files you should see that all incoming requests are coming from 127.0.0.1. To fix this you need to install mod_rpaf, the reverse proxy add forward module for Apache.

    # apt-get install libapache2-mod-rpaf


check content of /etc/apache2/mods-enabled/rpaf.conf :

<IfModule mod_rpaf.c>
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
</IfModule>


restart apache:

    # /etc/init.d/apache2 restart

Apache configuration (behind Nginx)

Nginx now acts as the front-end web server – waiting for requests on port 80 – you need to configure Apache to listen on a different port (8080 in this case) and preferably only on localhost, open the file /etc/apache2/ports.conf and change the line Listen 80 to Listen 127.0.0.1:8080, if you use name-based virtual hosts you should have a line NameVirtualHost *:80 in the same file. Change that to NameVirtualHost *:8080.

If you have configured Keep-Alive support in Apache you should disable it since it is already enabled in Nginx. Change KeepAlive On to KeepAlive Off in /etc/apache2/apache2.conf . You can also disable the mod_deflate module since Nginx already provides gzip compression.

nginx referer denial

In /etc/nginx/nginx.conf there is a list of words to deny in URLs. If URL contains these words, all referred links will not load. This causes missing images and stylesheets, and every link from that page to another on the same site will come up blank.

  ## Deny certain Referers (case insensitive)
  ## The ~* makes it case insensitive as opposed to just a ~
  if ($http_referer ~* (babes|...|zippo) ) {
        return 444;
     }

Just remove a word if you notice a problem and restart nginx with /etc/init.d/nginx restart

References

WordPress, MediaWiki, phpBB tweaks

WordPress

WordPress Nginx proxy cache integrator Enables your blog to work properly with an nginx frontend static proxy cache. ASTRONOMICAL performance is yours!

MediaWiki

Fix for mediawiki rewrite problem - double /docs/docs for example.

       $wgServer = 'http://dangerousprototypes.com';

The right setting in LocalSettings.php.

       $wgServer = 'http://dangerousprototypes.com/docs/'

This solution did not work for us.

phpBB

Fixing incorrect cookie settings in phpBB

  • Hint: to let users login after the update without clearing cookies, change the name of the phpBB3 cookie in the administrative control panel.

Hardening References

sysctl.conf security

http://www.cyberciti.biz/faq/linux-kernel-etcsysctl-conf-security-hardening/

tmp directory hardening

RootKit hunter

CSF firewall & LFD - Anti-DOS, Brute force detection and prevention, Port scan detection and prevention, Root logger

Securing SSH server

Log analysis

FTP hardening

Apache mod_security

Monitoring

AWS EC2 micro(Tokyo)- Remote Nagios and Munin server

Pingdom

References

Nagios
Munin

Backups!!!

Second HDD synchronization

Database dump

Offsite encrypted backups - Jungle Disk

Offsite encrypted backups on AWS EC3 or Rackspace Cloud Files

Troubleshooting