mem_to_axi: enforce store->fill ordering at the AXI boundary - #404
Open
ATWeatherly wants to merge 1 commit into
Open
mem_to_axi: enforce store->fill ordering at the AXI boundary#404ATWeatherly wants to merge 1 commit into
ATWeatherly wants to merge 1 commit into
Conversation
AXI orders neither reads after writes nor writes carrying different IDs, and VX_mem_to_axi issues stores with no completion tracking: awid was the requester's tag (so every store could take a different ID) and the B channel was discarded outright. A read-miss fill could therefore legally overtake an in-flight same-line store and return the pre-store data, which the cache then serves until eviction. Because the dcache is write-through, this surfaces as silently lost stores: the write reaches memory, but a stale line is already cached over it. We reproduced this on a Xilinx U50 (HBM). Kernel output stores go missing once the working set exceeds L1, with onset at a payload of about half the L1 size and the threshold scaling exactly 4x when L1 is scaled 4x (16KB L1: onset 2000 floats; 64KB L1: 8000). All 349 diagnosed losses in that campaign were kernel stores, none were host transfers, and a DMA soak against an idle core was clean over 86M words. Corrupted values were always the previous iteration's value at that address rather than a random bit pattern, and a later re-read still saw the stale value -- a dropped update, not a data-path fault. Fix, local to the adapter where the ordering is lost: - all writes on a bank share AWID 0. Same-ID writes are the only writes AXI requires the slave to keep in order, and in-order B responses let the tracker below retire FIFO-style; - a per-bank FIFO (WRITE_TRACK_DEPTH, default 8) holds issued-but- unacknowledged write addresses, pushed on the write handoff and retired on BVALID; - a read at the bank head is not issued while its address matches a tracked write, and a write may only start when a FIFO slot is free. A write that has already fired one of its two channels is never stalled, since its slot was reserved when it started; - the xbar pop is gated identically to the valids. axi_write_ready tracks the READY pins only, so an ungated pop would drop a stalled write from the xbar without it ever reaching AXI. This covers the cache write-through path and the non-cacheable/bypass path alike. Host-DMA vs kernel ordering runs over a separate AXI master and is out of scope here. Validated on hardware on the equivalent pre-refactor adapter: the U50 payload sweep that previously lost elements from 2000 floats upward ran 60/60 clean with zero lost elements, a kernel that previously corrupted 9 of 19 runs at 250MHz ran 20/20 with a bit-exact instruction count, and the cost was within noise (+0.02% cycles, timing still met). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01849vvnRzkZuWNY2B7RJGZr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This has been killing me when running llama2 benchmark. Causes lots of corruption to matmul's.
AXI orders neither reads after writes nor writes carrying different IDs, and VX_mem_to_axi issues stores with no completion tracking: awid was the requester's tag (so every store could take a different ID) and the B channel was discarded outright. A read-miss fill could therefore legally overtake an in-flight same-line store and return the pre-store data, which the cache then serves until eviction. Because the dcache is write-through, this surfaces as silently lost stores: the write reaches memory, but a stale line is already cached over it.
Reproduced this on a Xilinx U50 (HBM). Kernel output stores go missing once the working set exceeds L1, with onset at a payload of about half the L1 size and the threshold scaling exactly 4x when L1 is scaled 4x (16KB L1: onset 2000 floats; 64KB L1: 8000). All 349 diagnosed losses in that campaign were kernel stores, none were host transfers, and a DMA soak against an idle core was clean over 86M words. Corrupted values were always the previous iteration's value at that address rather than a random bit pattern, and a later re-read still saw the stale value -- a dropped update, not a data-path fault.
Fix, local to the adapter where the ordering is lost:
This covers the cache write-through path and the non-cacheable/bypass path alike. Host-DMA vs kernel ordering runs over a separate AXI master and is out of scope here.
Validated on hardware on the equivalent pre-refactor adapter: the U50 payload sweep that previously lost elements from 2000 floats upward ran 60/60 clean with zero lost elements, a kernel that previously corrupted 9 of 19 runs at 250MHz ran 20/20 with a bit-exact instruction count, and the cost was within noise (+0.02% cycles, timing still met).