HarnessGen
Some methods for generating harness/driver of library automatically.
Paper
- A Case Study on Automated Fuzz Target Generation for Large Codebases.ESEM 2019
Abstract: Fuzz Testing is a largely automated testing technique that provides random and unexpected input to a program in attempt to trigger failure conditions. Much of the research conducted thus far into Fuzz Testing has focused on developing improvements to available Fuzz Testing tools and frameworks in order to improve efficiency. In this paper however, we instead look at a way in which we can reduce the amount of developer time required to integrate Fuzz Testing to help maintain an existing codebase. We accomplish this with a new technique for automatically generating Fuzz Targets, the modified versions of programs on which Fuzz Testing tools operate. We evaluated three different Fuzz Testing solutions on the codebase of our industry partner and found a fully automated solution to result in significantly more bugs found with respect to the developer time required to implement said solution. Our research is an important step towards increasing the prevalence of Fuzz Testing by making it simpler to integrate a Fuzz Testing solution for maintaining an existing codebase.
- APICRAFT: Fuzz Driver Generation for Closed-source SDK Libraries.Usenix 2021
Abstract: Fuzz drivers are needed for fuzzing libraries. A fuzz driver is a program which can execute library functions by feeding them with inputs provided by the fuzzer. In practice, fuzz drivers are written by security experts and the drivers’ quality depends on the skill of their authors. To relieve manual efforts and ensure test quality, different techniques have been proposed to automatically generate fuzz drivers. However, existing techniques mostly rely on static analysis of source code, leaving the fuzz driver generation for closed-source SDK libraries an open problem. Fuzz driver generation for closed-source libraries is faced with two major challenges: 1) only limited information can be extracted from the library; 2) the semantic relations among API functions are complex yet their correctness needs to be ensured. To address these challenges, we propose APICRAFT, an automated fuzz driver generation technique. The core strategy of APICRAFT is collect – combine. First, APICRAFT leverages both static and dynamic information (headers, binaries, and traces) to collect control and data dependencies for API functions in a practical manner. Then, it uses a multi-objective genetic algorithm to combine the collected dependencies and build high-quality fuzz drivers. We implemented APICRAFT as a fuzz driver generation framework and evaluated it with five attack surfaces from the macOS SDK. In the evaluation, the fuzz drivers generated by APICRAFT demonstrate superior code coverage than the manually written ones, with an improvement of 64% on average. We further carried out a long-term fuzzing campaign with the fuzz drivers generated by APICRAFT. After around eight month’s fuzzing, we’ve so far discovered 142 vulnerabilities with 54 assigned CVEs in macOS SDK, which can affect popular Apple products such as Safari, Messages, Preview and so on.
- DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing.AAAI 2019
Abstract: Compilers are among the most fundamental programming tools for building software. However, production compilers remain buggy. Fuzz testing is often leveraged with newlygenerated, or mutated inputs in order to find new bugs or security vulnerabilities. In this paper, we propose a grammarbased fuzzing tool called DEEPFUZZ. Based on a generative Sequence-to-Sequence model, DEEPFUZZ automatically and continuously generates well-formed C programs. We use this set of new C programs to fuzz off-the-shelf C compilers, e.g., GCC and Clang/LLVM. We present a detailed case study to analyze the success rate and coverage improvement of the generated C programs for fuzz testing. We analyze the performance of DEEPFUZZ with three types of sampling methods as well as three types of generation strategies. Consequently, DEEPFUZZ improved the testing efficacy in regards to the line, function, and branch coverage. In our preliminary study, we found and reported 8 bugs of GCC, all of which are actively being addressed by developers.
- FUDGE: fuzz driver generation at scale.ESEC/FSE 2019
Abstract: At Google we have found tens of thousands of security and robustness bugs by fuzzing C and C++ libraries. To fuzz a library, a fuzzer requires a fuzz driver—which exercises some library code—to which it can pass inputs. Unfortunately, writing fuzz drivers remains a primarily manual exercise, a major hindrance to the widespread adoption of fuzzing. In this paper, we address this major hindrance by introducing the Fudge system for automated fuzz driver generation. Fudge automatically generates fuzz driver candidates for libraries based on existing client code. We have used Fudge to generate thousands of new drivers for a wide variety of libraries. Each generated driver includes a synthesized C/C++ program and a corresponding build script, and is automatically analyzed for quality. Developers have integrated over 200 of these generated drivers into continuous fuzzing services and have committed to address reported security bugs. Further, several of these fuzz drivers have been upstreamed to open source projects and integrated into the OSS-Fuzz fuzzing infrastructure. Running these fuzz drivers has resulted in over 150 bug fixes, including the elimination of numerous exploitable security vulnerabilities.
- FuzzBuilder: automated building greybox fuzzing environment for C/C++ library.ACSAC 2019
Abstract: Fuzzing is an effective method to find bugs in software. Many security communities are interested in fuzzing as an automated approach to verify software security because most of the bugs discovered by fuzzing are related to security vulnerabilities. However, not all software can be tested by fuzzing because fuzzing requires a running environment, especially an executable. Notably, in the case of libraries, most of the libraries do not have a relevant executable in practice. Thus, state-of-the-art fuzzers have a limitation to test an arbitrary library. To overcome this problem, we propose FuzzBuilder to provide an automated fuzzing environment for libraries. FuzzBuilder generates an executable that calls library API functions to enable library fuzzing. Moreover, any executable generated by FuzzBuilder is compatible with existing fuzzers such as AFL. We evaluate the overall performance of FuzzBuilder by testing open source libraries. Consequently, we discovered unknown bugs in libraries while achieving high code coverage. We believe that FuzzBuilder helps security researchers to save both setup cost and learning cost for library fuzzing.
- FuzzGen: Automatic Fuzzer Generation.Usenix 2020
Abstract: Fuzzing is a testing technique to discover unknown vulnerabilities in software. When applying fuzzing to libraries, the core idea of supplying random input remains unchanged, yet it is non-trivial to achieve good code coverage. Libraries cannot run as standalone programs, but instead are invoked through another application. Triggering code deep in a library remains challenging as specific sequences of API calls are required to build up the necessary state. Libraries are diverse and have unique interfaces that require unique fuzzers, so far written by a human analyst. To address this issue, we present FuzzGen, a tool for automatically synthesizing fuzzers for complex libraries in a given environment. FuzzGen leverages a whole system analysis to infer the library’s interface and synthesizes fuzzers specifically for that library. FuzzGen requires no human interaction and can be applied to a wide range of libraries. Furthermore, the generated fuzzers leverage LibFuzzer to achieve better code coverage and expose bugs that reside deep in the library. FuzzGen was evaluated on Debian and the Android Open Source Project (AOSP) selecting 7 libraries to generate fuzzers. So far, we have found 17 previously unpatched vulnerabilities with 6 assigned CVEs. The generated fuzzers achieve an average of 54.94% code coverage; an improvement of 6.94% when compared to manually written fuzzers, demonstrating the effectiveness and generality of FuzzGen.
- IntelliGen: Automatic Driver Synthesis for Fuzz Testing.ICSE-SEIP 2021
Abstract: Fuzzing is a technique widely used in vulnerability detection. The process usually involves writing effective fuzz driver programs, which, when done manually, can be extremely labor intensive. Previous attempts at automation leave much to be desired, in either degree of automation or quality of output. In this paper, we propose IntelliGen, a framework that constructs valid fuzz drivers automatically. First, IntelliGen determines a set of entry functions and evaluates their respective chance of exhibiting a vulnerability. Then, IntelliGen generates fuzz drivers for the entry functions through hierarchical parameter replacement and type inference. We implemented IntelliGen and evaluated its effectiveness on real-world programs selected from the Android Open-Source Project, Google’s fuzzer-testsuite and industrial collaborators. IntelliGen covered on average 1.08×-2.03× more basic blocks and 1.36×-2.06× more paths over state-of-the-art fuzz driver synthesizers FUDGE and FuzzGen. IntelliGen performed on par with manually written drivers and found 10 more bugs.
- RULF: Rust Library Fuzzing via API Dependency Graph Traversal.ASE 2021
Abstract: Robustness is a key concern for Rust library development because Rust promises no risks of undefined behaviors if developers use safe APIs only. Fuzzing is a practical approach for examining the robustness of programs. However, existing fuzzing tools are not directly applicable to library APIs due to the absence of fuzz targets. It mainly relies on human efforts to design fuzz targets case by case which is laborintensive. To address this problem, this paper proposes a novel automated fuzz target generation approach for fuzzing Rust libraries via API dependency graph traversal. We identify several essential requirements for library fuzzing, including validity and effectiveness of fuzz targets, high API coverage, and efficiency. To meet these requirements, we first employ breadth-first search with pruning to find API sequences under a length threshold, then we backward search longer sequences for uncovered APIs, and finally we optimize the sequence set as a set covering problem. We implement our fuzz target generator and conduct fuzzing experiments with AFL++ on several real-world popular Rust projects. Our tool finally generates 7 to 118 fuzz targets for each library with API coverage up to 0.92. We exercise each target with a threshold of 24 hours and find 30 previously-unknown bugs from seven libraries.
- SyRust: automatic testing of Rust libraries with semantic-aware program synthesis.PLDI 2021
Abstract: Rust’s type system ensures the safety of Rust programs; however, programmers can side-step some of the strict typing rules by using the unsafe keyword. A common use of unsafe Rust is by libraries. Bugs in these libraries undermine the safety of the entire Rust program. Therefore, it is crucial to thoroughly test library APIs to rule out bugs. Unfortunately, such testing relies on programmers to manually construct test cases, which is an inefficient and ineffective process. The goal of this paper is to develop a methodology for automatically generating Rust programs to effectively test Rust library APIs. The main challenge is to synthesize welltyped Rust programs to account for proper chaining of API calls and Rust’s ownership type system and polymorphic types. We develop a program synthesis technique for Rust library API testing, which relies on a novel logical encoding of typing constraints from Rust’s ownership type system. We implement SyRust, a testing framework for Rust libraries that automatically synthesizes semantically valid test cases. Our experiments on 30 popular open-source Rust libraries found 4 new bugs.
- WINNIE : Fuzzing Windows Applications with Harness Synthesis and Fast Cloning.NDSS 2021
Abstract: Fuzzing is an emerging technique to automatically validate programs and uncover bugs. It has been widely used to test many programs and has found thousands of security vulnerabilities. However, existing fuzzing efforts are mainly centered around Unix-like systems, as Windows imposes unique challenges for fuzzing: a closed-source ecosystem, the heavy use of graphical interfaces and the lack of fast process cloning machinery. In this paper, we propose two solutions to address the challenges Windows fuzzing faces. Our system, WINNIE, first tries to synthesize a harness for the application, a simple program that directly invokes target functions, based on sample executions. It then tests the harness, instead of the original complicated program, using an efficient implementation of fork on Windows. Using these techniques, WINNIE can bypass irrelevant GUI code to test logic deep within the application. We used WINNIE to fuzz 59 closed-source Windows binaries, and it successfully generated valid fuzzing harnesses for all of them. In our evaluation, WINNIE can support 2.2× more programs than existing Windows fuzzers could, and identified 3.9× more program states and achieved 26.6× faster execution. In total, WINNIE found 61 unique bugs in 32 Windows binaries.
Open-source Tools
gofuzzgen is generate template of fuzzing test code.
Go,Golang
The forkserver calls the target’s functionality under test through a harness. The harness is a dll specified in the AFL command-line and will get loaded by the fork-server. This harness is essentially responsible for acting as a shim between the forkserver and the target program.
C,C++
This repo contains a set of fuzzing harnesses for use with caffeine. It also has cmake configuration that makes compiling such a fuzzing harness easy.
C
This prototype is presented in USENIX 2021 as “APICraft: Fuzz Driver Generation for Closed-source SDK Libraries”.
Objective-C++
AutoHarness is a tool that automatically generates fuzzing harnesses for you. This idea stems from a concurrent problem in fuzzing codebases today: large codebases have thousands of functions and pieces of code that can be embedded fairly deep into the library. It is very hard or sometimes even impossible for smart fuzzers to reach that codepath. Even for large fuzzing projects such as oss-fuzz, there are still parts of the codebase that are not covered in fuzzing. Hence, this program tries to alleviate this problem in some capacity as well as provide a tool that security researchers can use to initially test a code base. This program only supports code bases which are coded in C and C++.
C,CodeQL
Generate a fuzzing harness for (shared) libraries from Binary Ninja.
Python
FuzzGen, is a tool for automatically synthesizing fuzzers for complex libraries in a given environment. FuzzGen leverages a whole system analysis to infer the library’s interface and synthesizes fuzzers specifically for that library. FuzzGen is fully automatic and can be applied to a wide range of libraries. The generated fuzzers leverage LibFuzzer to achieve better code coverage and expose bugs that reside deep in the library.
C,C++
Fuzzing is a great (pen)testing tool, as long as you have one or just a handful of functions you want to fuzz. Fuzzing libpng is no problem, but fuzzing something like OpenSSL is nigh-impossible because you need to manually write the fuzzing boilerplate for every single function. This is an attempt to make fuzzing libraries with large API surfaces feasible by auto-generating the boilerplate.
Rust
FuzzBuilder is a tool for generating an executable automatically for library fuzzing by using a unit test. Further, FuzzBuilder can generate seed files to fuzz a specific library API function by analyzing a unit test.
C,C++
Above the FuzzBuilder.
This prototype tool is aimed at generating fuzz targets for Rust libraries automatically. A fuzz target is a function that accepts an array of bytes and exercises some library APIs. The targets generated by our tool is a set of shallow fuzz targets (the length of each target is mostly smaller than or equal to 3) to cover as many library APIs as possible. Each generated target can be compiled successfully. Then, you can fuzz these targets with afl.rs to find potential bugs.
Rust
Some tools about fuzzing libraries
Sloth is a fuzzing setup that makes use of libFuzzer and QEMU’s user-mode emulation (qemu/linux-user) on x86_64/aarch64 host to emulate aarch64 Android libraries to fuzz the target Android native library.
C++,C
Differential cryptography fuzzing
Fuzz the inner functions in libraries.
Python