Fix sys_compat_mmap arithmetic bug

[Haibib]

PR Description

• Fixes an arithmetic bug in sys_compat_mmap() function where page sizes are compared against the requested number of bytes instead of the remaining number of bytes that need to be mapped.
• Added a test to verify correct mapping, rounding up to the nearest 4KB page without over-allocating
• Added explicit checks to panic if unexpected kernel behavior is encountered

[keithw]

What goes wrong in the current code? Is it possible to write a test for this? (How did this bug even come up?)

[Haibib]

What goes wrong in the current code? Is it possible to write a test for this? (How did this bug even come up?)

With the current code, if 3 MB is requested and the address pointer p is a multiple of 2 MB, the loop checks that the requested length is above 2 MB and maps a 2 MB page. After incrementing p by 2 MB, it erroneously checks the requested length again instead of checking the remaining bytes. It maps a second 2 MB page rather than covering the last 1 MB with 4 KB pages. I stumbled upon this bug when I tried to run "just fix eval addblob.fix" in the fix-parser branch. Compared to the other small examples, this might've been the first time that some requested bytes exceeded 2 MB, triggering this bug.

[keithw]

Sure, but, like, how practically does this come up? It sounds like the kernel is erroneously mapping more memory than it has to -- is this perceivable by the fix shell (or the Fix procedure, or by the user running the command at the Linux command-line)? Does something go wrong if we run just fix eval addblob.fix without applying this change? Or is there a good way to see all the memory mappings to visually see that there is a bug here?

[keithw]

I guess my other question would be, are we sure that if the process asks for 3 MiB, we definitely want the kernel to map "one 2 MiB page + 256 4 KiB pages"?

E.g. as opposed to "two 2-MiB pages," or "768 4-KiB pages"?

It wasn't totally obvious to me that one of these three choices is the right one... there seems to be some compromise between tolerance for overallocating vs. the descriptive complexity of the mapping (and therefore the coded length of the continuation). But I guess you're saying something like, "yes, the policy should be (or already is, but there was a bug?) that 'one 2-MiB page + 256 4-KiB pages' is the way to go, and if the process doesn't like how big the page table ends up being, it should have rounded up its own request to 4 MiB itself"?

[Haibib]

Sure, but, like, how practically does this come up? It sounds like the kernel is erroneously mapping more memory than it has to -- is this perceivable by the fix shell (or the Fix procedure, or by the user running the command at the Linux command-line)? Does something go wrong if we run just fix eval addblob.fix without applying this change? Or is there a good way to see all the memory mappings to visually see that there is a bug here?

Yes, running just fix eval addblob.fix in the fix-parser branch causes a crash with the following error:

[2026-09-02T21:47:42Z ERROR kernel::types::function] exited with exception: PageFault { addr: a2c4ac, error: 6 } @ rip=0x7f0000aaba
KERNEL PANIC: panicked at fix/src/runtime/arca.rs:54:21:
unexpected non-effect return
----- BACKTRACE -----
...

[keithw]

Ah, I didn't realize it was crashing! But if the problem is that the kernel was mapping too much memory, then... why is there an unexpected page fault? (Should we throw in some debug_asserts that p has advanced by (a) at least the size of the requested allocation, and (b) less than 4 KiB beyond the size of the requested allocation? And if we did this, would the first of those asserts have been failing without this fix, i.e. is there some scenario where the current code doesn't allocate enough?)

[Haibib]

I guess my other question would be, are we sure that if the process asks for 3 MiB, we definitely want the kernel to map "one 2 MiB page + 256 4 KiB pages"?

E.g. as opposed to "two 2-MiB pages," or "768 4-KiB pages"?

It wasn't totally obvious to me that one of these three choices is the right one... there seems to be some compromise between tolerance for overallocating vs. the descriptive complexity of the mapping (and therefore the coded length of the continuation). But I guess you're saying something like, "yes, the policy should be (or already is, but there was a bug?) that 'one 2-MiB page + 256 4-KiB pages' is the way to go, and if the process doesn't like how big the page table ends up being, it should have rounded up its own request to 4 MiB itself"?

The current policy seems to be trying to map the largest possible pages and fill the rest with 4 KiB pages (the "one 2 MiB page + 256 4 KiB pages" option), and this PR is just fixing a bug in that mechanism. At least currently, I think the function is only being called from the fix shell. If we always control the calling point, I lean toward making the caller decide whether to round up and the kernel exactly fulfill the explicit request. We can find a more sophisticated policy that maintains determinism, but from a cursory search, a simple policy is to promote to a larger page if utilization exceeds some empirically derived threshold, as described in this paper: Coordinated and efficient huge page management with ingens. I'm not sure if @Akshay-Srivatsan has thoughts on this.

[Haibib]

Ah, I didn't realize it was crashing! But if the problem is that the kernel was mapping too much memory, then... why is there an unexpected page fault? (Should we throw in some debug_asserts that p has advanced by (a) at least the size of the requested allocation, and (b) less than 4 KiB beyond the size of the requested allocation? And if we did this, would the first of those asserts have been failing without this fix, i.e. is there some scenario where the current code doesn't allocate enough?)

We can add in the debug_asserts. Assert B would have failed without this fix, but I don't think assert A would have failed because it loops until the entire requested length is mapped. I think the page fault occurs because after the over-allocation, the next request tries to grow its memory starting inside the over-allocated 2 MiB page and tries to map 4 KiB pages in the same region. Then, arca/src/table.rs's map() function tries to get a table to map those 4 KiB pages, but since there's a 2 MiB page there instead of a table, it creates a table for that 2 MiB span and drops the original 2 MiB page. Then, when the code tries to access the memory mapped by the original 2 MiB page, there is a page fault error for attempting to access an unmapped page.

[keithw]

Per live discussion, it sounds like there might be three bugs here:

• Kernel is mapping too much (what's fixed so far in this PR)
• On a subsequent sys_compat_mmap call that targets an overlapping range, the kernel is silently unmapping a 2 MiB page from the old mapping and therefore leaving an unmapped area that eventually triggers the page fault (my opinion is that it should probably fail with an error instead, because trying to communicate the semantics of what got unmapped (given our use of different page sizes) seems way too complicated)
• the fix shell / wasm-rt implementation was not looking at the return value from arca_compat_mmap to see if the mmap succeeded and/or if the return value is what it expected (my opinion is that it should also probably just fail with an error if there is an overallocation or failure)

[Haibib]

Per live discussion, it sounds like there might be three bugs here:

• Kernel is mapping too much (what's fixed so far in this PR)
• On a subsequent sys_compat_mmap call that targets an overlapping range, the kernel is silently unmapping a 2 MiB page from the old mapping and therefore leaving an unmapped area that eventually triggers the page fault (my opinion is that it should probably fail with an error instead, because trying to communicate the semantics of what got unmapped (given our use of different page sizes) seems way too complicated)
• the fix shell / wasm-rt implementation was not looking at the return value from arca_compat_mmap to see if the mmap succeeded and/or if the return value is what it expected (my opinion is that it should also probably just fail with an error if there is an overallocation or failure)

Yes, I've added explicit checks to fail with an error when encountering unexpected kernel behavior.