building IRFuzzer

[regehr]

hello, I'd like to try out IRFuzzer but when I run ./build.sh I'm getting an error, below.

also, is there any version of IRFuzzer that works with LLVM top of tree? all of the fuzzing that I do is against the latest version. thanks!

....
clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04/libexec/ccc-analyzer
clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04/libexec/c++-analyzer
clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04/libexec/analyze-c++
clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04/libexec/analyze-cc
clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04/libexec/intercept-c++
clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04/libexec/intercept-cc
fatal: not a git repository (or any of the parent directories): .git
regehr@ohm:~/IRFuzzer$  

[DataCorrupted]

Hi John! Thanks for trying our tool! Unfortunately, I'm travelling this week. I will get back to you next week.

[regehr]

no rush.
I'll give a bit of background-- we're doing LLVM backend testing using a fork of Alive2 that can do translation validation for the AArch64 backend. we're finding plenty of bugs using our own fuzzer, but the more fuzzers the better! we'd love to try out yours if possible, and of course we would be happy to tag you in any bugs that we found using the help of your tool.

[DataCorrupted]

I heard that from Yoyo in LLVM Developer's Meeting 2022. Validating backend is way harder than finding ICE, we are pushing that direction as well. Maybe we should schedule a talk or smt :)

[regehr]

that sounds great, let's talk sometime in March

[regehr]

I was thinking a little more about how we could work together. one way is we build your fuzzer and then have it call our tools. but, also, if you want to run our translation validation tool, then that is easy too. I would think it would just plug into your workflow pretty seamlessly. basically you just hand it an IR function and it either tells you that it verified or else signals some sort of error.

I'd be happy to help you get started using our software if you wanted to do that. it's all easy stuff. all I would ask is that you let us know about any miscompiles that you find, since we're still accumulating evidence that our stuff works well, in preparation for writing it up for publication.

so far we've found and reported 29 silent miscompiles in LLVM's AArch64 backend. I'm sure there are a lot more bugs remaining but we need fuzzer magic to discover them.

[DataCorrupted]

We have found some bugs, still deduplicating/ping pointing them. Our experiment runs on X86 backend, so we can compile it against our host machine directly without the need of an emulator, run it, and see the difference between O0 and O3.

[DataCorrupted]

git clone https://github.com/SecurityLab-UCD/IRFuzzer.git -b irfuzzer-0.3
cd IRFuzzer
./init.sh
./build.sh

I separated the initialization and building process. Please try this to reproduce the results we had in our paper.
Upgrading to the latest LLVM turned out to be harder than I thought, will you ping you once that it's done (probably in IRFuzzer 0.4).
The reason is that we have to customize LLVM for some of our needs to track the TableGen, and that part isn't being nice when bumped to the latest LLVM. :(

Besides, I don't think IRFuzzer 0.3 is a good fit for your need. IRFuzzer is known to generate UB (e.g., store undef), the situation is worsened after we introduce architecture agonistic intrinsic insertion. It gives you compilable, but not likely to be runnable code that may not be verifiable.

Internally, we have a verify-asm branch that is customized to generate only defined behavior for that purpose. We should definitely talk about your needs, and maybe we can customize them for you as well. :)

[regehr]

hi Peter, thanks!! I'll try this out soon.

my testing workflow is based on Alive2 and it is extremely robust with respect to undefined behavior. even so, I think it would be best if you avoided undef because it is going to go away and also the LLVM people are very reluctant to fix miscompiles that are triggered by undef. but poison and immediate UB are perfectly fine.

[regehr]

well, it got pretty far this time, here's what I ended up with on an x86-64 machine running Ubuntu 22.04:

[3455/3569] Linking CXX executable bin/llvm-cov
[3496/3569] Linking CXX executable bin/llvm-profgen

[3569/3569] Linking CXX executable bin/llvm-reduce
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- The ASM compiler identification is unknown
-- Didn't find assembler
CMake Error at CMakeLists.txt:47 (project):
  The CMAKE_C_COMPILER:

    clang

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.


CMake Error at CMakeLists.txt:47 (project):
  The CMAKE_CXX_COMPILER:

    clang++

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.


CMake Error at CMakeLists.txt:47 (project):
  No CMAKE_ASM_COMPILER could be found.

  Tell CMake where to find the compiler by setting either the environment
  variable "ASM" or the CMake cache entry CMAKE_ASM_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.


-- Warning: Did not find file Compiler/-ASM
-- Configuring incomplete, errors occurred!
See also "/home/regehr/IRFuzzer/llvm-project/build-release/CMakeFiles/CMakeOutput.log".
See also "/home/regehr/IRFuzzer/llvm-project/build-release/CMakeFiles/CMakeError.log".
regehr@ohm:~/IRFuzzer$ 

[regehr]

(the build above was using the system compiler, gcc 11.4.0. before that, I tried building using clang 17, but in that case it didn't get nearly as far)

[regehr]

anyway, I'm happy to wait for IRFuzzer 0.4

like I said, I do not require UB-free code, I just need functions in LLVM IR that can be processed by the AArch64 backend. if llc -march=aarch64 foo.ll can process the file, then it is likely to work for me

[DataCorrupted]

• Ideally, we are using LLVM 14 to compile everything. init should have installed LLVM14, but I just realized I forgot to tell build where it is. Fixed already. Should work now.
• If it still doesn't work on your host machine for some reason, I created a docker file based on ubuntu 22.04. cd IRFuzzer && docker build . should do it.
• Or if none of these works, let me know and I'll update a built image to docker.io. I didn't want to do it as the image is super large: IRFuzzer builds debug builds of LLVM - it is huge.

As of IRFuzzer 0.4, I think I will trim off some features just so IRFuzzer can direct rely on LLVM instead of our customized LLVM so that updating would be easier. (I think calling it IRFuzzer-Alive2 would be more appropriate lol).

I'm bogged down by ICSE 25, job hunting and some other projects, I don't have an estimated time to finish, but you can expect it by next Sunday I guess.

[regehr]

hi Peter, thanks!
of course no rush on this. we already have a fuzzer to combine with our tools, but yours looks great and we hopefully can find more bugs using it!! :)

[DataCorrupted]

@regehr branch irfuzzer-alive is available now. Since you mentioned you are testing AArch64, this branch only compiled that backend (You can modify build.sh to change that). Feel free to let me know if you need anything else.

Looking back at your comment about undef and poison, I am tempted to change how IRFuzzer generates these.