1.46.0 fails to build with llvm-libunwind enabled

[Keruspe]

On one of my systems which is entirely based on llvm/clang/libc++/llvm-libunwind 1.46.0 builds fine, but on the other "traditional" ones, it fails with

error: linking with `/usr/host/bin/x86_64-pc-linux-gnu-cc` failed: exit code: 1                                                                                                                                                                                                                                                
  |                                                                                                                                                                                                                                                                                                                            
  = note: "/usr/host/bin/x86_64-pc-linux-gnu-cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0
-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-7b7ea85adaabb772.rustc_binary.4nmnfy3t-cgu.0.rcgu.o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-7b7e
a85adaabb772.rustc_binary.4nmnfy3t-cgu.1.rcgu.o" "-o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-7b7ea85adaabb772" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" 
"/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-r
ust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/jemalloc-sys-741a979a41c097fb/out/build/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/psm
-434df6e9150bc0ba/out" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-37977d896f0c1d8a/out" "-L" "/usr/x86_64-pc-linux-gnu/lib/llvm/10/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc
-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps" "-lrustc_driver-ca92058eee3505a8" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libjemalloc_sys-40622d79400e9af9.rlib" "-Wl,--start-group" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-lstd-230cc113f5bd99f4" "-Wl,--end-group" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-5c27ddf29299677e.rlib" "-Wl,-Bdynamic" "-lz" "-lrt" "-ldl" "-ltinfo" "-lpthread" "-lm" "-lxml2" "-lstdc++" "-lpthread" "-lutil" "-ldl" "-lutil" "-ldl" "-lrt" "-lpthread" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"                                                                                                                              = note: /usr/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld: /var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ca92058eee3505a8.so: undefined reference to `_Unwind_Resume'                                
          /usr/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld: /var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ca92058eee3505a8.so: undefined reference to `_Unwind_GetIP'                                 
          /usr/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ld: /var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ca92058eee3505a8.so: undefined reference to `_Unwind_Backtrace'                             
          collect2: error: ld returned 1 exit status      

1.45.2 compiles fine.

I'm currently doing some testing but I suspect that 21abc88 is the culprit

[Keruspe]

I can confirm that reverting #72746 fixes the build

[tmandry]

Can you provide more details about how you're building Rust? Are you using out-of-tree LLVM?

[Keruspe]

I package rust for Exherbo (source-based) and I build it using its package manager.
For stable rust, we use the system llvm by default indeed

[Keruspe]

@tmandry How can we get this to move forward?

I don't know how to reproduce the initial issue to find another fix but the current one is either wrong on incomplete given this failure

[petrhosek]

The use case that was addressed by #72746 is mixing C++ and Rust code in a single process. In those cases, we don't want libunwind that's linked into Rust binaries to preempt libunwind symbols of C++ binaries because the two unwinders may be incompatible. This is particularly problematic when libunwind used by C++ is sanitized but Rust one isn't which leads to runtime failure that we observed in Fuchsia.

The solution is to build libunwind as a hermetic library to avoid re-exporting its symbols. We already use -fvisibility=hidden when building libunwind but that doesn't take effect unless we disable visibility annotations which is what #72746 does.

I haven't managed to reproduce this failure locally, but my guess is that in your build librustc_driver doesn't explicitly link unwinder and instead relies on unwinder symbols being provided transitively by some other library. After #72746, those symbols are no longer exported hence the link failure. If that's indeed the case, then I think the correct solution isn't to revert that change but to find out why unwinder isn't being linked explicitly.

[Keruspe]

Could this be somehow related to the use of -Wl,--as-needed?

[Keruspe]

It is not, just retries disabling it.

The failing linker invocation is:

"/usr/host/bin/x86_64-pc-linux-gnu-cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/l
ib" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-43d4d556549614b1.rustc_binary.98yjbizw-cgu.0.rcgu.o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc
-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_binary-43d4d556549614b1.rustc_binary.98yjbizw-cgu.1.rcgu.o" "-o" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rust
c/x86_64-unknown-linux-gnu/release/deps/rustc_binary-43d4d556549614b1" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-
unknown-linux-gnu/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-ru
stc/x86_64-unknown-linux-gnu/release/build/jemalloc-sys-a57df08f1cff1043/out/build/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/psm-4c77fdac541dea43/out" "-L" 
"/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-3d5b9a19a4407b23/out" "-L" "/usr/x86_64-pc-linux-gnu/lib/llvm/10/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-
1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps" 
"-lrustc_driver-45869c97924aea2d" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libjemalloc_sys-10e304a877f3c25b.rlib" "-Wl,--start-group" "-L" "/var/tmp/pa
ludis/build/dev-lang-rust-1.46.0/work/rustc-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-lstd-9779a050e626c3f3" "-Wl,--end-group" "-Wl,-Bstatic" "/var/tmp/paludis/build/dev-lang-rust-1.46.0/work/rustc
-1.46.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib64/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-6758141eee354fd1.rlib" "-Wl,-Bdynamic" "-lz" "-lrt" "-ldl" "-ltinfo" "-lpthread" "-lm" "-lxml2" "-lstdc++" "-lpthread" "-lutil" "-ldl" "-lutil" "-ldl" "
-lrt" "-lpthread" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"

Looks like rustc adds some --as-needed even if it's not in LDFLAGS.

Let's get the interesting part out of this:

"-lstd-9779a050e626c3f3" "-lz" "-lrt" "-ldl" "-ltinfo" "-lpthread" "-lm" "-lxml2" "-lstdc++" "-lpthread" "-lutil" "-ldl" "-lutil" "-ldl" "-lrt" "-lpthread" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"

I don't see any "-lunwind" in the linker arguments, or am I missing something here?

[Keruspe]

I see in the logs libunwind-32bae9364f36982a.rmeta being mentioned during builds of panic_unwind, and libunwind-32bae9364f36982a.rlib being mentioned during builds of std.
-lunwind never appears.

@petrhosek when and how is this linkage supposed to happen?

[Keruspe]

Inside libstd.so I see the two "missing" symbols:

00000000000acf70 l     F .text  00000180 _Unwind_Backtrace
00000000000ae280 l     F .text  00000064 _Unwind_GetIP

[mati865]

@Keruspe can you look for the same symbols in librustc_driver*.so?

[Keruspe]

0000000000000000         *UND*  00000000 _Unwind_Backtrace
0000000000000000         *UND*  00000000 _Unwind_GetIP

[mati865]

@petrhosek IIUC before #72746 when rust.llvm-libunwind in the config.toml was enabled, whole libunwind was reexported by libstd-<hash>.so (this option disables linking unwind library from the system) which is not essentially correct.
We reexport necessary symbols for our use here so librustc_driver_<hash>.so should have those symbols provided by libstd-<hash>.so.
Does this sound right?

On one of my systems which is entirely based on llvm/clang/libc++/llvm-libunwind 1.46.0 builds fine, but on the other "traditional" ones, it fails

I haven't managed to reproduce this failure locally

My gut says it's BFD vs LLD difference but I cannot test it soon.

[Keruspe]

Testing with lld

[Keruspe]

Same error with lld and bfd.

I've come to the same understanding as you, fwiw. And since rustc_driver doesn't list the unwind crate in its dependencies, I fail to see how it could get the symbols the same way std does.

Maybe when building rust with the system llvm and with llvm-libunwind enabled we should dynamically link to the system llvm-libunwind instead?