Last week I got a new box for doing heavy data pipeline development at $DAYJOB: a MinisForum MS-A2 with the Ryzen 9955HX. This was a bit overdue given that my MacBook is now over 5 years old and has approximately half the space I really need to work on some projects. After a bit of deliberation and number crunching, it was clear that buying was significantly more attractive than renting. Even in this crazy market where DDR5 accounted for more than half the total cost, buying would break even vs a similarly spec'd rented server (which didn't even have DDR5) in just 12.5 months! Given that I fully expect a work machine to last 5 years or so, it was a no brainer.

And then I remembered everything I disliked about working with x86 consumer hardware from by brief stint in IT 🫠

Tremors

On Friday afternoon, I started noticing some... odd behaviors. I admit I'm currently working on fairly taxing data pipelines built using Apache DataFusion, but I was noticing an odd escalation of strange behaviors, which accumulated over a period of several days.

• ZFS warning logs where the system detected a corrupted write via checksumming.
• A surprisingly invalid result from a processing job, which would nondeterministically fail. Codex curiously identified as differing from the expected value by a single bit flip. Spidey sense activated!
• Increasing general system instability... like `pkg`` checksum failures 😱

Maybe it's RAM?

Given the bit flip in particular, and everything I'd heard about DDR5 being a bit more fragile and the on-die "ECC" being worthless, I assumed I was unlucky and got a bad batch of RAM.

So I got copies of MemTest86+ and Memtest86 Pro loaded up on USB drives. The commercial variant didn't find any issues, but the open-source one lit up when it hit the AVX2 tests. Online reports seemed to suggest that the open-source Memtest86+ was not well designed for modern systems, but it's hard to tell how much of this is motivated by... other things...

So I reseated the RAM and shuffled sticks around a few times. Same results. I just so happened to have a workload that could knock it over, which was lucky since most testing tools I found either didn't repro it or were so degenerate that they caused system instability in other ways, which I was not super interested in.

Maybe it's FreeBSD??

In a vain attempt to rationalize things, at this point I wondered if I had hit a FreeBSD kernel bug, incompatibility issue with the Zen 5 CPUs (Ryzen support had been flaky in the early days), or something along those lines. AMD and Intel are famously terrible at providing documentation, so a lot is reverse engineered by the community.

Wait, What is Even Knowable?!

And then I went to an even darker place... Did I hit a compiler bug in my data pipeline?! I found this Rust miscompilation bug which mentions a similar CPU, but has been around for a LONG time...

But how do I know if my kernel is what I think it is? Or that the compiler binary isn't corrupt? Fortunately my system design with verifiable packages (pkg check -s -a) and ZFS made that fairly straightforward.

SIMD? Power? Firmware?

Memtest86 showed the RAM was fine. But Memtest86+ didn't... One of the things that stood out to me was that Memtest86 didn't spend very much time on several sorts of tests which had a lot of potential to indirectly test the CPU.

By coincidence (maybe), the first test that exposed issues under Memtest86+ was an AVX2 test. I don't know why neither project supports the higher vector extensions like AVX512, but this struck me as interesting. So I tried recompiling my program without AVX512. No dice. In fact, at this point, I started hitting even worse issues, like a SIGBUS in the middle of a compilation.

By this point, a few interesting facts had surfaced, and begun swirling around in my head...

• These Ryzen 9 chips run quite hot and have a bit of a history of stability issues.
• Undervolting and bespoke cooling seem to be fairly commonplace, but neither are really feasible with the limited controls of the MinisForum BIOS or case design.
• The vendor-provided firmware has a history of bugs. The most recent fuss being related to the 1.03 update breaking PCIe passthrough for virtualized setups (e.g. Proxmox) and widespread reports of bricked devices if you downgrade.
• This post about thermal settings, which would pop up in basically every third search.

Unstable at any... nevermind 🤣

I had read that last post about a "temperature fix" several times, but wrote it off because ti came across as rather wrong-headed in its framing.

1. I don't care how quiet or loud the machine is if it's performing well.
2. Increasing your CPU life by "800%" with this one weird trick comes off as obviously misguided.

But it did seem lime the MinisForum BIOS (which literally nobody has anything good to say about) might be overdoing things in the defaults. Given that there are no voltage controls (at least not without overclock settings, which I have absolutely no interest in touching), it was worth a shot.

Switching the power mode back to "balanced" (from performance) AND adding the 78 degree temperature limit seemed to return the system to stability. I have not observed any performance drop as a result of the change. In fact, it seems like it's actually faster for real workloads (e.g. a 20 minute long killer process of mine now runs in 15 minutes).

I'm pretty annoyed at MinisForum and AMD, but I guess I shouldn't be too surprised. It's like a trope now that most x86 hardware is plagued by terribly written, proprietary firmware. That the MinisForum BIOS can, in a default configuration, result in system instability is unacceptable.

Given that the temperatures in the failing test were not at all unreasonable, I can only assume that the "performance" power profile is responsible for the instability. I'll still keep the TjMax capped at 78 anyways, since it keeps fans quieter and actually kept a real workload moving faster.

Hopefully by the time I need a new desktop, we'll have more options for serious ARM hardware. And maybe in the interim we'll get a BIOS / firmware fix from MinisForum.

While this post is fairly negative, I should also say I think the physical construction of the box is EXCELLENT, and it's super easy to pop the guts out of the enclosure which is so nice coming from the NUC. And having multiple NICs is fantastic! But the BIOS issues are really inexcusable.