In August I posted a memory-corruption report here and then withdrew it, because my mitigation claim turned out to be a time confound rather than a measurement. I said I would only come back with something that survived its own null hypothesis. This is that report.
We run a small fleet of AXON V0.2 boards. Two of them silently corrupt memory: single bytes change value in resident pages, with no kernel message, no MCE, and nothing in dmesg. One instance reached a build artefact on disk and was caught only because our build re-hashes every component after copying it. One board is still faulting as I write this.
Evidence bundle attached (39 KB — logs, analysis, and the tooling sources, so you can reproduce this rather than trust us).
Hardware: AXON V0.2, 8 GiB LPDDR4X, 7,926 MiB usable, no ECC. Kernel 6.1.75.
We know this is non-ECC RAM. That is exactly why we measured before posting.
We are not reporting “a bit flipped once”. Ambient soft errors are real and we expect a few. What we cannot reconcile with ambient soft errors is the rate and the structure:
| ambient soft error | what we actually see | |
|---|---|---|
| bits per event | overwhelmingly single-bit | never single-bit — always 2 or 4 (17/17 captures) |
| spatial distribution | uniform over the array | 611× enriched into 0.53% of the address space |
| byte alignment | none | every faulting word 32-byte aligned; faults confined to one byte lane |
| rate | rare | 172 independent events on one board in ~3 days |
The 611× figure is measured against a control, not assumed. Clustering is only meaningful if you know where the page cache actually landed — so we mapped 1 GiB, walked /proc/self/pagemap, and found the cache spanning 8,072 MiB of physical space with 0.161% of it inside the fault window. Expected frames in the window if faults were positional coincidence: 0.242. Observed: 148 of 150. (04-pagecache-control.txt)
The two boards
| Board A | Board B | |
|---|---|---|
| instrument | page-cache vs O_DIRECT detector |
768 MiB mlocked hash-verified canary |
| exposure | 4,412 passes, ~3 days | 1,275 passes, ~3.5 days (still running) |
| independent events | 172 | 17 captures in 2 events |
| bad bytes | 2,045 | 46 bits across 17 words |
| frames implicated | 150 (600 KiB = 0.0074% of RAM) | 3 (12 KiB = 0.00015%) |
| localisation | 148/150 inside 41.95 MiB | 3 frames inside 30.5 MiB |
| bit direction | 96% SET (0→1) | 96% CLEARED (1→0) |
| burst offsets | 8, 10, 12, 14 | 4, 6 |
| damage unit | 50 single frames + 50 adjacent pairs, never 3+ | one adjacent pair + one single |
Both boards fault in the same /proc/iomem bank (100000000–1ffffffff), in different regions of it. Opposite bit polarity on the two boards is, we think, the signature of two independent device faults rather than one shared design issue — we mention it because it argues against the simplest “it’s a platform bug” reading.
Two findings we think matter most:
- Corruption follows the frame, not the data. 20 physical frames corrupted multiple unrelated file pages; no file page was ever corrupted in two different frames (172 events, 172 distinct file pages). This is a property of the memory, not of any file or workload.
- It is transient, not a stuck cell. On Board B: re-read three times → same wrong value (17/17), so the wrong value was genuinely stored. Rewrite the word → truth restored (17/17).
sticky: false17/17. This is why a memtester pass comes back clean on a board that is actively corrupting data.
The DRAM frequency controller — what we chased, and what we retracted
This is the claim I withdrew in topic 530, restated here with the numbers that killed it — because DVFS is the first thing anyone will suggest, and I would rather nobody spend time on a rockchip-dmc lead I introduced.
Our original A/B was confounded by time: it compared “inside the fault window, governor unpinned” against “after the fault window, governor pinned”. The fault is time-clustered, so the pin took credit for the window closing on its own. Re-run properly:
- 87 iterations on stock
dmc_ondemand, 10,975 frequency transitions counted → zero faults. - The healthy control board switches 2.9× more than the corrupting one (99,492 vs 34,802 transitions).
- Board B’s DMC governor is pinned at
performance— 9 transitions in 4 days of uptime — and it produced all 17 captures anyway. The faults happen with the DRAM clock essentially static.
DRAM DVFS is exonerated. Whatever this is, frequency switching is not the trigger. (Temperatures at capture were 38.8–39.8 °C, so it is not thermal either.)
What this costs us
This is the part we would most like taken seriously. The problem is not that memory occasionally fails — it is that this platform gives us no way to know when it has.
- A build artefact was corrupted and written to disk. One byte,
0xF3→0x72, at offset 4,021,543 of a 13.7 MB binary. File size unchanged, every non-cryptographic check passed. It was caught only by our own digest verification. We do not know how many times this happened before we started checking. - Every transfer now costs four reads. We digest every component after copy, at both ends of every transfer, at extraction, and on the booted slot.
- Two bespoke instruments run permanently, consuming real capacity on boards we paid for compute with.
- Both boards are restricted to development work. Neither is trusted to produce anything of record.
- There is no hardware reporting at all.
CONFIG_EDAC_ROCKCHIP=yis built in butrk_edacnever binds — zero devices under/sys/devices/system/edac/mc/, and 0 of 5,224dmesglines mention EDAC or ECC.CONFIG_MEMORY_FAILUREis not set, so there is no runtime frame retirement either. We cannot retire 600 KiB of known-bad RAM without a DTreserved-memoryoverlay we write by hand. - We cannot certify a board as healthy. Our nightly sweep reaches 96% of RAM and reports zero hits — on a board that then corrupts something hours later. Coverage and exposure are different axes, and a wide sweep buys the first at the cost of the second.
What we ruled out, and how
| suspect | how it was eliminated |
|---|---|
| storage / NVMe | 0 error/reset/timeout lines; source file hashes 7/7 correct warm and cold |
| filesystem | ext4 clean, 0 EXT4-fs errors |
| our copy scripts | install(1) from a source that verifies correct; 30/30 later copies byte-perfect |
| thermals | 38.8–39.8 °C at capture |
| DRAM DVFS | 10,975 transitions → zero faults; healthy board switches 2.9× more |
| instrument artefact | fault injection OFF (inject:0 on all 8 starts); canary asserts its own VmLck — memtester-class tools silently shrink their buffer when mlock fails |
| sampling bias | we read the canary’s source: both traversal branches visit every word, so it was free to report other offsets and never did |
Our deviations, stated up front
Custom kernel 6.1.75-axon-fleet4 (vendor 6.1 tree, our config), our own rootfs, NVMe root rather than eMMC, CPU and DMC governors pinned to performance (for an unrelated inference-throughput reason, measured separately). We are happy to reproduce on a stock image if that helps — please say so and we will.
One honest caveat rather than a tidier story: the direction of the byte-lane confinement is not stable across capture epochs. Every fault we can currently re-verify sits at an even burst offset (the low byte, DQ0–7), but two older records — one of them the corrupted build artefact — sit at odd offsets. The claim we stand behind is that faults are confined to one byte lane of the 16-bit word at a time, never both. Which lane has varied, and we cannot explain that. Details in 13-build-artefact-corruption.txt.
Questions
- Is even-byte-offset / single-DQ-lane confinement a known signature on V0.2 — a marginal lane, a write-leveling margin, or a DDR training result?
- Are the DDR training parameters adjustable from TPL, and is there a way to dump the trained per-channel/per-lane values on a running board? That would let us compare a corrupting board against a healthy one directly.
- Is
rk_edacexpected to bind on RK3588? If so, what DT node does it need? Is there any in-band memory error reporting on this SoC? - Has Vicharak seen byte-level corruption in this batch? We have removed serial numbers from the public bundle — happy to send SoC / eMMC / NVMe serials privately if they enable a batch check or RMA.
- Is there a recommended production memory qualification beyond memtester? Ours passes a 96%-coverage sweep and still corrupts.
- For the interim: is a DT
reserved-memory+no-mapcarve-out the recommended way to retire known-bad frames, or is there a vendor path we have missed?
Happy to run any test you suggest on either board — both are instrumented and one is actively faulting right now.
axon-v02-dram-corruption-evidence.zip (38.8 KB)