Saturday, April 16, 2016

A cheap and effective Web App Firewall with continuos real time attack monitoring. #nginx #mod_security #naxsi #ElasticSearch #Kibana

I wanted to share one of my projects I worked on last year. We were trying to solve for “how to alert real time Web attacks” on our infrastructure. After a lot of brainstorming sessions we discarded the idea of having enterprise WAF solutions which are sold by many big players in the market. Most of the enterprise solutions work “inline” with your internet traffic and that chokes up considerable amount of your bandwidth.Having said that, I do not discourage anyone exploring these solutions. 

So we started exploring multiple open source products which are under active development. We chose NAXSI and Mod Security as our prime targets and started our research in how we can extract the best out these two.

Since we were experimenting these on Nginx web server, we had to evaluate the one which gives us desirable output in terms of minimal performance impact and almost no false positives. We evaluated ModSecurity first and found it to be quite unstable, we observed multiple nginx worker processes dying on a regular intervals. However, these problems might have been solved with current commits to the project [https://github.com/SpiderLabs/ModSecurity/issues].
I found NAXSI to be more stable in terms of performance, but it requires a lot of tuning to cut down false/positives.

Stats:[This may vary depending upon what all modules nginx has been compiled]
Nginx -
65K qps
Max CPU Usage: 55%
Nginx+Naxsi
65K qps
Max CPU Usage: 68%

So, here on, I will be talking about how one can compile NAXSI with Nginx 1.4.4+ and fine tune it and have a continuous alert monitoring around the same.

What is NAXSI?:

NAXSI means Nginx Anti Xss & Sql Injection.Its a web application firewall (WAF) which comes as a nginx module which needs to be compiled from source, it is also available as a package for many UNIX-like platforms. This module, by default, reads a small subset of simple rules (naxsi_core.rules) containing 99% of known patterns involved in websites vulnerabilities. For example, '<', '|' or 'drop' are not supposed to be part of a URI.
Naxsi is different from other WAF solutions, since it totally relies on a whitelist approach and not a signature based approached which is a lot more slower and resource consuming.

So with Naxsi in place we were able to have decent picture of which all IP addresses were attacking us and how we can stop them at our edge network.


To start with let's look at a simple architecture of our Web application Firewall.


Screen Shot 2016-04-15 at 8.28.18 PM.png


Here are simple steps to compile NAXSI from source :
  1. Select the nginx tar file here(http://nginx.org/download/)
  2. Untar it in the /opt folder
  3. wget the newest naxsi source code (https://github.com/nbs-system/naxsi) in /opt folder
  4. install libpcre3-dev
  5. cd nginx-1.4.4 and start compiling
  6. sudo ./configure --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --with-http_ssl_module --with-debug --http-log-path=/var/log/nginx/access.log --conf-path=/etc/nginx/fk_nginx.conf --with-http_stub_status_module --user=nginx --error-log-path=/var/log/nginx/error.log --prefix=/usr/local/nginx_new --sbin-path=/usr/sbin/nginx --with-http_realip_module --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --add-module=../naxsi/naxsi_src/
  7. sudo make && sudo make install
  8. Now copy the naxsi main ruleset file to /etc/nginx → sudo cp naxsi_core.rules /etc/nginx/naxsi_core.rules
  9. sudo nano /etc/nginx/nginx.conf
  10. [add the /etc/nginx/naxsi_core.rules in include directive] 
sudo nano/etc/nginx/nginx.conf
http {

       include /etc/nginx/naxsi_core.rules;

       }
11. Create the Whitelist file my_naxsi.rules
Few links for whitelist creation:
https://github.com/nbs-system/naxsi/wiki/whitelists
For a simple apt-get installation follow : https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-naxsi-on-ubuntu-14-04

We used NAXSI in learning mode to avoid as much noise as possible. For creating the ruleset I recommend running it in production in Learning mode and gathering a significant amount of valid traffic, and post running the nxutil script to generate the whitelist that is specific to your prod environment.
More details on nxutils can be found here: https://github.com/prajal/nxutil
 
NAXSI write all its attack logs in nginx error location:
/var/log/nginx/error.log

Example Error logs:
2015/03/27 12:00:18 [error] 22909#0: *13840757 NAXSI_FMT: ip=ATTACKIP&server=www.yourcompany.com&uri=/&learning=1&vers=0.54&total_processed=22184&total_blocked=20&block=1&zone0=BODY&id0=16&var_name0=, client: ATTACKIP, server: www.youcompany.com, request: "POST /?t=12:00:17%20PM HTTP/1.1", host: "A.B.C.D"

2015/11/14 14:43:36 [error] 5182#0: *10 NAXSI_FMT: ip=X.X.X.X&server=Y.Y.Y.Y&uri=/some--file.html&learning=1&total_processed=10&total_blocked=6&zone0=URL&id0=1007&var_name0=&zone1=ARGS&id1=1007&var_name1=asd, client: X.X.X.X, server: localhost, request: "GET /some--file.html?asd=-- HTTP/1.1", host: "Y.Y.Y.Y"

Now that we have the log, we need to ingest it into our Log system. The most important part of this process is indexing the correct components we would want to visualise later in kibana. For us the crucial part was the clientIP “ip” and the “request” part of the log.




Screen Shot 2016-04-16 at 11.59.29 AM.png

Now once your Kibana dashboard is up and ready I recommend using Elastalerts  
[https://github.com/Yelp/elastalert] to get the necessary alerting for the attack IP’s that you are monitoring for.

Here is a quick attack alert triggered:






Some References:
If you are planning to implement NAXSI in your infrastructure I recommend reading