Skip to content

mem_to_axi: enforce store->fill ordering at the AXI boundary - #404

Open
ATWeatherly wants to merge 1 commit into
vortexgpgpu:masterfrom
ATWeatherly:fix/axi-store-order
Open

mem_to_axi: enforce store->fill ordering at the AXI boundary#404
ATWeatherly wants to merge 1 commit into
vortexgpgpu:masterfrom
ATWeatherly:fix/axi-store-order

Conversation

@ATWeatherly

Copy link
Copy Markdown
Contributor

This has been killing me when running llama2 benchmark. Causes lots of corruption to matmul's.

AXI orders neither reads after writes nor writes carrying different IDs, and VX_mem_to_axi issues stores with no completion tracking: awid was the requester's tag (so every store could take a different ID) and the B channel was discarded outright. A read-miss fill could therefore legally overtake an in-flight same-line store and return the pre-store data, which the cache then serves until eviction. Because the dcache is write-through, this surfaces as silently lost stores: the write reaches memory, but a stale line is already cached over it.

Reproduced this on a Xilinx U50 (HBM). Kernel output stores go missing once the working set exceeds L1, with onset at a payload of about half the L1 size and the threshold scaling exactly 4x when L1 is scaled 4x (16KB L1: onset 2000 floats; 64KB L1: 8000). All 349 diagnosed losses in that campaign were kernel stores, none were host transfers, and a DMA soak against an idle core was clean over 86M words. Corrupted values were always the previous iteration's value at that address rather than a random bit pattern, and a later re-read still saw the stale value -- a dropped update, not a data-path fault.

Fix, local to the adapter where the ordering is lost:

  • all writes on a bank share AWID 0. Same-ID writes are the only writes AXI requires the slave to keep in order, and in-order B responses let the tracker below retire FIFO-style;
  • a per-bank FIFO (WRITE_TRACK_DEPTH, default 8) holds issued-but- unacknowledged write addresses, pushed on the write handoff and retired on BVALID;
  • a read at the bank head is not issued while its address matches a tracked write, and a write may only start when a FIFO slot is free. A write that has already fired one of its two channels is never stalled, since its slot was reserved when it started;
  • the xbar pop is gated identically to the valids. axi_write_ready tracks the READY pins only, so an ungated pop would drop a stalled write from the xbar without it ever reaching AXI.

This covers the cache write-through path and the non-cacheable/bypass path alike. Host-DMA vs kernel ordering runs over a separate AXI master and is out of scope here.

Validated on hardware on the equivalent pre-refactor adapter: the U50 payload sweep that previously lost elements from 2000 floats upward ran 60/60 clean with zero lost elements, a kernel that previously corrupted 9 of 19 runs at 250MHz ran 20/20 with a bit-exact instruction count, and the cost was within noise (+0.02% cycles, timing still met).

AXI orders neither reads after writes nor writes carrying different IDs,
and VX_mem_to_axi issues stores with no completion tracking: awid was the
requester's tag (so every store could take a different ID) and the B
channel was discarded outright. A read-miss fill could therefore legally
overtake an in-flight same-line store and return the pre-store data,
which the cache then serves until eviction. Because the dcache is
write-through, this surfaces as silently lost stores: the write reaches
memory, but a stale line is already cached over it.

We reproduced this on a Xilinx U50 (HBM). Kernel output stores go missing
once the working set exceeds L1, with onset at a payload of about half
the L1 size and the threshold scaling exactly 4x when L1 is scaled 4x
(16KB L1: onset 2000 floats; 64KB L1: 8000). All 349 diagnosed losses in
that campaign were kernel stores, none were host transfers, and a DMA
soak against an idle core was clean over 86M words. Corrupted values were
always the previous iteration's value at that address rather than a
random bit pattern, and a later re-read still saw the stale value --
a dropped update, not a data-path fault.

Fix, local to the adapter where the ordering is lost:

 - all writes on a bank share AWID 0. Same-ID writes are the only writes
   AXI requires the slave to keep in order, and in-order B responses let
   the tracker below retire FIFO-style;
 - a per-bank FIFO (WRITE_TRACK_DEPTH, default 8) holds issued-but-
   unacknowledged write addresses, pushed on the write handoff and
   retired on BVALID;
 - a read at the bank head is not issued while its address matches a
   tracked write, and a write may only start when a FIFO slot is free.
   A write that has already fired one of its two channels is never
   stalled, since its slot was reserved when it started;
 - the xbar pop is gated identically to the valids. axi_write_ready
   tracks the READY pins only, so an ungated pop would drop a stalled
   write from the xbar without it ever reaching AXI.

This covers the cache write-through path and the non-cacheable/bypass
path alike. Host-DMA vs kernel ordering runs over a separate AXI master
and is out of scope here.

Validated on hardware on the equivalent pre-refactor adapter: the U50
payload sweep that previously lost elements from 2000 floats upward ran
60/60 clean with zero lost elements, a kernel that previously corrupted
9 of 19 runs at 250MHz ran 20/20 with a bit-exact instruction count, and
the cost was within noise (+0.02% cycles, timing still met).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01849vvnRzkZuWNY2B7RJGZr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant