Reverse proxy for GAE application using Nginx and SSL

If you use Google App Engine you probably already know that you can use a custom domain mapped to your application server on the appspot.com domain. Unfortunately, the custom domain works only when accessed access via http. If you want to use SSL with the custom domain, you’re out of luck.

Until GAE adds support for custom SSL certificates, your best bet is to use a reverse proxy in front of your appspot.com application. For me, as a developer, messing up with SSL certificates and web server/proxy configuration sounded somewhat complicated and time consuming but it all turned out to be quite simple.

I assume in this post that you’re moderately familiar with Linux shell script and have a server access to install and configure your reverse proxy. If you already use a custom domain for your GAE application, you’ll have to change your DNS settings to point to your reverse proxy server’s IP address instead to ghs.google.com. (You don’t have to remove domain from Goolge yet.)

Why Nginx?

Well… I started using Apache at first. It comes as a part of the default Linux distribution and we already run our web sites on it. There’s enough information about settings up Apache as a reverse proxy and that was not an issue. However, once our Apache-based reverse proxy was up and running it was slow… depressingly slow. The first HTTP request (that opens connection to web server and includes SSL handshake) was taking as much as 5 seconds. The subsequent requests (within keep-alive time) were fast, as long as the connection was alive. I was somewhat discouraged by various claims that SSL adds a significant CPU overhead but fortunately this was not an issue…

Further research took me to consider Pound and Nginx as possible alternatives to Apache. Reading about experiences with these two and especially after reading this post about Nginx’s performance with SSL, I decided to go with Nginx. It turned out quite well!

Let’s start…

Get and install Nginx

Login to your future reverse proxy system and:

sudo apt-get install nginx

Get SSL certificate

Prices for SSL certificates vary. We took GoDaddy‘s Standard SSL (it’s $60 for 1 year although you can easily find a coupon and get it somewhat cheaper). After your certificate request has been verified, go and download certificate files. GoDaddy has no download for Nginx so choose Apache. You will get a ZIP containing two files:

1. <Common Name>.crt – Your certificate
2. gd_bundle.crt – GoDaddy Certificate Intermediates Bundle

(For more detailed explanation on obtaining certificate you may find this useful.)

Concatenate certificates

Nginx’s ssl_certificate directive accepts a single SSL certificate file (no alternative for Apache’s SSLCertificateChainFile) so you need to concatenate your site’s and GoDaddy’s chain crt file:

cat mysite.crt gd_bundle.crt > mysite_and_gd_bundle.crt

Remove passphrase from your key file

You should remove passphrase from your key file if you want Nginx to be able to startup without asking you for the key passphrase.

cp mysite.key mysite.key.bak
openssl rsa -in mysite.key.bak -out mysite.key

Reverse proxy configuration

It’s time to configure Nginx:

cd /etc/nginx/sites-available

create a new file named secure.mysite.com (using your site name obviously), copy and customize configuration:

server {
  listen 443;
  server_name secure.mysite.com;

  access_log /<path-to>/access.log;
  error_log  /<path-to>/error.log;

  keepalive_timeout 70;

  ssl  on;
  ssl_certificate     /<path-to>/mysite_and_gd_bundle.crt;
  ssl_certificate_key /<path-to>/mysite.key;
  ssl_session_timeout  30m;

  location / {
    proxy_pass       https://<my-gae-app-id>.appspot.com;
    proxy_set_header Host <my-gae-app-id>.appspot.com;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_intercept_errors on;
    error_page 500 = /error_page.html;
  }

  location = /error_page.html {
    root /local_path_to_static_files_root;
  }
}

Explanation:

• Listen port 443 only (for https). You may want to specify IP address too if your system has more than one IP address.
• Server name secure.mysite.com is an external server name for which we’ve obtained the certificate.
• We proxy_pass to our app on GAE using https. (You could use http too but I don’t recommend it as an overhead is really low. We didn’t see significant latency or increased CPU usage raising from using https here.)
• Set Host header to our app’s server name at GAE. Without it GAE will not know which application you’re looking for.
• Our application on GAE will see all requests coming from our proxy server’s IP address so we send additional header with user’s IP address.
• We intercept our app’s errors (line 22) and in case of error 500 (line 23) we show user a static page located on our proxy server (lines 26-28). This is finally a way to serve static page on GAE’s error 500.

Completing Nginx setup

Nginx will handle only SSL connections so disable its default site configuration (listening on port 80) and activate our site’s config:

cd /etc/nginx/sites-enabled
sudo rm default
sudo ln -s ../sites-available/secure.mysite.com .

Nginx is ready. One more thing…

Update Apache config

Apache is still listening on port 80 which means it will receive requests to http://secure.mysite.com. This is not what we want so update its config to redirect all requests to https:

<VirtualHost *:80>
  ServerName secure.mysite.com

  RewriteEngine On

  DocumentRoot /local_path_to_static_files_root

  <Directory "/local_path_to_static_files_root/">
    AllowOverride All
    Order allow,deny
    allow from all
    RewriteRule ^(.*) https:/secure.mysite.com/$1 [R,L]
  </Directory>

  Redirect permanent / https://secure.mysite.com/
</VirtualHost>

(I’m sure it’s not perfect but it worked for me.)

To make sure Apache is not listening on 443, disable its SSL module:

sudo a2dismod ssl
sudo /etc/init.d/apache force-reload

That’s it!

We’re ready to start Nginx:

sudo /etc/init.d/nginx start

and try to open https://secure.mysite.com

That’s it! We got a branded, secure URL for our application for about $50 (annually for SSL certificate) and few hours of administration time. We used our “regular” web server (it was idling anyway) to host Nginx and so far it’s coping with the load very well. (If we see it struggling at any time, we’ll be more than happy to upgrade it. As we don’t offer real Freemium, almost all of our users are customers so we’d probably be happy to have an issue with the proxy load.)

Additional considerations

• Many reverse proxy configuration samples that I’ve found use “proxy_redirect off” option. My application didn’t work well with it so I removed it (“on” is default). If you have issues with redirects you may want to check this option.
• Nginx supports additional options for SSL negotiation (ssl_protocols, ssl_ciphers, etc.) that may be used to fine tune SSL handshake performance. I still don’t fully understand all of these so I left it alone.
• You may want to set Nginx’s worker_processes to the number of processors your server has.

Using Java? Speed-up bug fixing and improve customer satisfaction using LogDigger to create detailed error reports and notifications for your web application

Tagged as: AppEngine, Nginx, Ubuntu