Comparing FuzzLabs to American Fuzzy Lop (AFL) is not a good idea as the two tools are very different. The way they work internally, the scope, the attack vectors, the whole approach are all different. However, I keep being asked how I would compare FuzzLabs to AFL; I thought it is time to discuss the topic a bit.

Test Case Generation

When it comes to mutation generation, AFL operates as a dumb fuzzer (or brute-force fuzzer — as per the documentation). It is a mutation-based fuzzer, which means you have a known, valid file continuously mutated. Of course, this approach has benefits, but it comes with severe limitations that I will discuss later. Overall, it is a fantastic tool to recommend to anyone doing fuzz testing to use wherever applicable.

FuzzLabs, on the other hand, is a general-purpose hybrid fuzzer capable of acting as a generation-based fuzzer, a mutation-based fuzzer, or the combination of the two at the same time. Unlike AFL, FuzzLabs is aware of the structure of the data it is working with due to the generation-based test generation capabilities. It is also aware of any relationship between the different fields within the structured data.

Feature FuzzLabs AFL
Data structure aware Yes No
Aware of field relationships Yes No
Testing languages such as HTML, SQL, and JavaScript Yes No
Can work with structured data containing checksums Yes No
Can work with structured data containing encryption or encrypted data Yes No
Can work with structured data containing compression or compressed data Yes No
Mutation generation capabilities comparison summary

Even though generation-based fuzzing comes with lots of benefits that make fuzzing much more efficient and fast, having to describe the structure of the data in a form that the fuzzer can understand scares people away. This process is historically known to be a slow and error-prone process requiring some level of understanding of the data structure. FuzzLabs addresses this issue, but I will not go into details as it is outside of this post’s scope.

In the case of AFL, users do not need to know the data structure; however, the data fed to the fuzzer has to be carefully selected. For example, suppose you are testing an application/library that processes PNG files. In that case, you have to ensure the PNG files you feed it with cover all possible permutations of the different PNG sections and that each is still a valid (!) image. Otherwise, if you want AFL to figure these out, a test run will take a very long time. Ultimately, what this means in terms of coverage is that in both case, you probably end up reading the relevant RFC’s, but:

  • In case of FuzzLabs, you define the image using the Visual Protocol Modeler or the Python API
  • In the case of AFL, you would have to find a way to generate the right input images first. How you do that, it’s up to you.

Attack Vectors

Even though AFL focuses on testing libraries that process files, some spin-offs of AFL can do more than that by now. The problem comes when you want it to do something more, or you want it to do something slightly different. It takes significant effort to make the necessary changes. An alternative, which is to collect all the various tools and try to integrate them in an easy-to-manage form, did not prove much easier.

FuzzLabs by design gives you a framework that can be extended in many ways. One of these directly relates to how FuzzLabs interfaces with the test targets. FuzzLabs supports multiple attack vectors. For example, file format fuzzing is as much an option as network fuzzing. If you wanted to, you could even test things over Bluetooth, a serial line, or CAN – without much trouble. Extending FuzzLabs is pretty much a piece of cake than any other fuzzers – and this is also one of the many positive feedback we have received so far.

 

Coverage

The comparison under this section is a bit off as the two products work in a very different way. Still, some may find this information useful, so here it is.

In case of AFL, the code coverage is determined by the input file(s), the performance of the genetic algorithm and time. The later one is basically the configuration on how long you let it generate random stuff to detect new paths before it gives up.

In the case of FuzzLabs, the coverage is dictated by the templates that describe structured data (sometimes also referred to as the “grammar”) and the mutation algorithms of each primitive. The primitives are the basic building blocks of a template and they are used to represent different field types. The better your template is and the more carefully you configure the primitives within the better your coverage will get.

There’s another thing that I initially wanted to discuss under the mutation capabilities. Still, I realized it makes much more sense to discuss here as it primarily relates to coverage. With FuzzLabs, you can build up states that influence test case generation. If you are looking to find vulnerabilities in media parsing libraries, this is not important. It gets critical as soon as you start looking into testing something that communicates using a sequence of messages: any protocols, be it over the network or not. These messages are often related to each other, or an application’s internal state, or both. The simplest example would be testing a protocol that implements authentication. Much time will be wasted for nothing if the fuzzer cannot extract authorization tokens that it must submit with subsequent requests, or it is not aware whether the authentication was successful or not in the first place. The fuzzer/users must be able to:

  • Process and use data returned by the target
  • Define what should and what must not be mutated
  • Define blocks that should or should not be rendered and mutated within the structured data based on certain conditions

These capabilities improve coverage and save you from time wasted on generating test cases that do not add value.

Issue Triaging

Crash triage. It’s all about the crashes. When it comes to quality assurance, crashes are not the only bad things that can happen. While FuzzLabs can be used to detect crashes during test runs, its triage features are more limited when it comes to crash triaging. Most of the triaging have to be done manually based on the information provided within the reports which includes:

  • The information provided by the GDB monitor extension: the location of the fault (assembly instructions), register values, stack dump.
  • The most relevant subset of the mutation(s) that were sent to the target prior the crash.

However, FuzzLabs can be used to detect other issues, as well. Analyzing the connection state, processing log files to e.g., catch exceptions, or detecting issues from the console output, perform response-based processing via defined rules, detect spikes in CPU/Memory utilization, or a combination of these are all possible ways to detect unexpected/unintended behaviour.

Summary

It is never one or the other. Picking the right fuzzer(s) always boils down to the goal and the most efficient way to reach that goal. I recommend everyone to do the research required to make an informed decision.

.