Although Web Application Firewalls (WAFs) are recognized as an effective aspect of a defense in depth strategy, there are few tools that attempt to objectively review their effectiveness. Research companies like NSS or Gartner perform benchmarks of WAFs, but their methodologies are rarely disclosed. With the advent of site reliability and devops cultures, infrastructure as code has been a strategy to verify functionality of products. This talk brings that same mentality to WAFs; not only do we verify WAF functionality within deployments, but we also provide a method to verify WAF defenses against new exploits and attacks. We do this with our project FTW - Framework for Testing WAFs.
We achieved two outcomes from this project. The first was to design a framework that is extendable to test arbitrary WAF implementations. This would allow engineers to compare WAFs to help them make an informed purchase decision for their organization instead of relying on reports and literature that do not disclose their testing methodologies. Secondly, we want to have the ability to develop new tests without the need for development experience. This allows rapid prototyping of attack payloads without the need of a scripting language. These payloads are then executed against various WAF implementations to see how the WAF responds. Once tested, new rules can be deployed within the WAF and then the attack is added to a corpus of attacks for continuous testing.
We will first review the design of the tests. We use the OWASP Core Ruleset Version 3 (CRSv3) as our benchmark for web attacks and defenses, so the first task was to translate the CRS and write attacks to make sure the rules trigger. This resulted in a corpus of 1000s of attacks provided for end users at no cost. Tests are written in YAML format, and we will go into detail on how the format is developed to include both basic HTTP attacks as well as more advanced multi-stage attacks.
Next, we will review the architecture of the code. Py.test is used as the testing foundation due to the wide adoption within industry it enjoys, its ability to parametrize the YAML test files, as well as its ease of use in continuous integration environments. We show how an individual can set up an FTW testing environment and start writing or editing tests, as well as creating new ones. We will then show continuous integration strategies to test and deploy new WAF rules. We use Travis-CI as the continuous integration technology, but traditional CI or deployment tech can also be used.
We then will move into the crux of our presentation where we highlight the results. We plan to discuss how this project is being used throughout the community. The ModSecurity team used FTW extensively for regression testing in the CRS. We will show lessons learned and how regression testing in security is extremely important. We will also show a use case for how an origination uses FTW to ship WAF rules for their customers on the edge. Strategies to ship WAF rules include continuous integration and applying security to the SDLC of these deployments. Lastly, we highlight a journaling feature that allows security engineers and red teamers issue a battering ram of web attacks and log responses into a local database for pentest reports.
The Current Code
https://github.com/crs-support/ftw to check our code
https://github.com/fastly/waf_testbed for a VM that spawns the latest CRS w/ the latest FTW to start running tests
https://github.com/SpiderLabs/OWASP-CRS-regressions/tree/master/tests for CRS attacks
https://github.com/SpiderLabs/owasp-modsecurity-crs/ latest CRS
