rygorous · HackerTrans

2 ปีที่แล้วdiscuss

Yeah, pushing current CPUs (Intel and AMD both) as far as they'll go is well into the diminishing returns. For AMD HW I'd likewise recommend using one of the "Eco" modes. The single-digit percentage points you get out of those last few hundred MHz really don't move the needle on productivity workloads, and the power draw reduction is substantial. It also makes the machines much quieter under load.

rygorous

2 ปีที่แล้วdiscuss

It's an issue that, while the mainboard is involved, happens to occur on (at least) the 3 best-selling mainboard vendors compatible with that family of CPUs, at stock settings, so that you can take an affected CPU, swap it through a selection of the most popular mainboards compatible with said CPU and see the same kind of instability problems.

I don't think it's unreasonable to call that Intel's problem, maybe not in terms of culpability (but truly, nobody cares) but definitely in the sense this is doing damage to their brand. If the mainboards are all out of spec then they need to talk about this publicly, rein them in, start a certification program, whatever. Being publicly completely fine with this as long as it results in good review scores but then starting to go "well actually..." when there's stability issues on a small fraction of sold units is not a good look.

rygorous

2 ปีที่แล้วdiscuss

That error comes out of Unreal Engine proper, not Oodle, and is meant to be shown in cases when compressed shader data is corrupted in the game data files on disk in a way that hasn't been caught by package validation. Which is to say, it's not exactly common.

Oodle itself has some diagnostics hooked up to logs but none of that is user-facing. All the user-facing stuff needs to get handed off multiple times to get from low-level IO plumbing to somewhere that even knows how to display a user-facing error message to begin with.

The most common cause for that error message was, and continues to be (except on the relatively small number of affected machines), that compressed shader data on disk is corrupted. If and when we have a handle as to what actually causes the problem and a minimally-invasive fix (as opposed to the list of several different anecdotal "what if you try X?" that we got from Intel HW lab folks), we'll try to detect affected machines and point them to a website with instructions. For now, it's just a random error message that happens to frequently show up on machines encountering this issue, and all that would change if we changed the error message was that we'd confuse end users more and have to list two different error messages on that page instead of one.

rygorous

2 ปีที่แล้วdiscuss

Thanks!

Chance, sure, it's just a matter of logistics. Revision is a bit tricky since it's usually shortly after GDC, a very busy time in the game engine/middleware space I work in, so not usually when I feel up to a pair of international flights. :) Best odds are for something between Christmas and New Year's Eve since that's when I'm usually in Germany visiting family and friends anyway.

rygorous

2 ปีที่แล้วdiscuss

(Oodle maintainer here.) This issue only occurs on some small fraction of machines, but on those that we've had access to, it reproduces with BIOS defaults and no user-specified overclocking. It turns out several of these mainboards will overclock and set other values out of spec even at BIOS defaults.

I don't have a problem with end users experiencing instability once they manually overclock (that's how it goes), but CPU + mainboard combinations experiencing typical OC symptoms with out-of-the-box settings is just not OK.

This appears to be an arms race between mainboard vendors all going further and further past spec by default because it gives better benchmark and review scores and their competition does it. Intel for their part are themselves also dialing in their parts more aggressively (and, presumably although I don't know for sure, with smaller margins) over time, and they are for sure aware that this is happening, because a) even had they not known already (which they did) they would have learned about this months ago when we first contacted them about this issue, b) technically out of spec or not, as long as it seems to work fine for users and makes their parts look better in reviews, they're not going to complain.

However, it turns out, it does not work fine for at least some small fraction of machines. I have no idea what that percentage is, but it's high enough that googling for say "Intel 13900K crash" yields plenty of relevant results. Some of this will be actual intentional overclockers but, given how boards default to some extend of out-of-spec overclocking enabled, it's unlikely to be all of them.

Meanwhile we (and other SW vendors) are getting a noticeable uptick in crash reports on, specifically, recent K-series Intel CPUs, and it's not something we can sanely work around because the issue manifests as code randomly misbehaving and it's not even when doing anything fancy. The Oodle issue in particular is during LZ77-family decompression, which is to say, all integer arithmetic (not even multiplies, just adds, shifts and logic ops), loads/stores and branches. This is the bare essentials. If it was an issue with say AVX2, we could avoid AVX2 code paths on that family of machines (and preferably figure out what exactly is going wrong so we can come up with a more targeted workaround). But there is no sane plan B for "integer ALU ops, load/stores and branches don't work reliably under load". If we can't rely on that working, there is not enough left for us to work around bugs with!

I realize this all looks like finger-pointing, but this is truly beyond our capacity to work around in a sane way in SW, with what we know so far anyway. Maybe there is a much more specific trigger involved that we could avoid, but if so, we haven't found it yet.

Either way, when it's easy to find end user machines that are crashing at stock settings, things have gone too far and Intel needs to sit down with their HW partners and get everyone (themselves included) to de-escalate.

rygorous

2 ปีที่แล้วdiscuss

Oodle maintainer here, we had two people that hit the issue offer to run some experiments for us. Neither were doing any overclocking before and both tried numerous things including resetting to BIOS defaults and also updating their BIOS (there was a known [to Intel] issue affecting some ASUS boards that had been fixed in a BIOS update in spring of 2023, and we were asked to rule it out.)

This issue doesn't affect every such machine, but both people affected by the issue that consented to run tests for us still had the issue reproduce after flashing BIOS to current and with BIOS default settings for absolutely everything.

Among the settings enabled by default on some boards: current limit set to 511 amps (...wat), long duration power limit set to 350W (Intel spec: 125W), short duration power limit also set to 350W (Intel spec: 253W), "MultiCore Enhancement" which is extra clock boosting past what the CPUs do themselves set to "Auto" not "Off", and some others.

rygorous

2 ปีที่แล้วdiscuss

Definitely not. These are supposed to be higher-quality bins that also ship with higher stock clock rates (both base and boost) and are rated for them.

I don't know how common this is across the whole population of PC buyers, but personally, I have for sure bought K-series parts then not clocked them past their stock settings, trusting that they are rated for it and deeply uninterested in any OCing past that. (I prefer my machines stable, thank you very much.)

rygorous

2 ปีที่แล้วdiscuss

This is blowing up now, but the first report of this kind of issue that reached me (I'm the current Oodle maintainer) was in spring of last year. We've been trying to track it down (and been in contact with Intel) since then. The page linked in the OP has been up since December.

Epic Games Tools is B2B and we don't generally get bug reports from end users (although later last year, we did have 2 end users write to us directly because of this problem - first time this has happened for Oodle that I can think of, and I've been working on this project since 2015). Point being, we're normally at least one level removed from end user bug reports, so add at least a few weeks while our customers get bug reports from end users but haven't seen enough of them yet to get in touch with us (this is a rare failure that only affects a small fraction of machines).

13900Ks have been out since late Oct 2022. It's possible that this doesn't show up on parts right out of the box and takes a few months. It's equally plausible that it's been happening for some people for as long as they've had those CPUs, and the first such customers just bought their new machines late 2022, maybe reported a bug around the holidays/EOY that nobody looked at until January, and then it took another 2-3 months for 3-4 other similar crashes to show up that ultimately resulted in this case getting escalated to us.

rygorous

2 ปีที่แล้วdiscuss

Granny, Iggy and Miles are all discontinued as stand-alone products. We're still providing support to existing customers but not selling any new licenses.

rygorous

2 ปีที่แล้วdiscuss

(I'm Oodle maintainer and did most of this investigation.)

For the majority of systems "in the wild", I don't know. We had two people with affected machines contact us and consent to do some testing for us, and in both cases the issue still reproduced after resetting the BIOS settings to defaults.

rygorous

2 ปีที่แล้วdiscuss

Dynamic switching power (i.e. the fraction of the chip's power consumption from actually switching transistors, as opposed to just "being on") scales with V^2 * f, where V=voltage and f=frequency, and V in turn depends on f, where higher frequencies need higher voltage. Not really linearly (It's Complicated(tm)), but it's not a terrible first-order approximation, which makes the dynamic switching power have a roughly cubic dependency on frequency.

Therefore, 1.1x the frequency at the high end (where switching power dominates) is 1.33x the power draw.

Those final few hundred MHz really hurt. Conversely, that's also why you see "Eco" power profiles with a major reduction in power draw that cost you maybe 5-10% of your peak performance.

rygorous

2 ปีที่แล้วdiscuss

(I'm the person who did most of the investigation.)

A relatively major realization during the investigation was that a different mystery bug that also seemed to be affecting many Unreal Engine games, namely a spurious "out of video memory" error reported by the graphics driver, seemed to be occurring not just on similar hardware, but in fact the exact same machines.

For a public example, if you google for "gamerevolution the finals crash on launch" and "gamerevolution the finals out of video memory", you'll find a pair of articles describing different errors, one resulting from an Oodle decompression error, and one from the graphics driver spuriously reporting out-of-memory errors, both posted on the same day with the same suggested fix (lower P-core max clock multiplier).

That's the problem right there in a nutshell. It's not just Oodle detecting spurious errors during its validation. Other code on the same machine is glitching too. And "just try repeating" is not a great fix because we can't trust the "should we repeat?" check any more on that machine than we can trust any of the other consistency checks that we already know are spuriously failing at a high rate.

Many known HW issues you can work around in software just fine, but frequent spurious CPU errors don't fall into that category.

rygorous

2 ปีที่แล้วdiscuss

Absolutely, yes.

It can also misbehave without any hardware bugs due to glitching. Rates of incidence of this must be quite low or that would be considered a HW bug, but it's never zero. Run code for enough hours on enough machines collecting stack traces or core dumps on crashes and you will notice that there's a low base rate of failures that make absolutely no sense. (E.g. a null pointer dereference literally right after a successful non-null pointer check 2 instructions above it in the disassembly.)

You will also notice that many machines in a big fleet that log such errors do so exactly once and never again, but some reoccur several times and have a noticeably elevated failure rate even though they're running the exact same code as everyone else. This too is normal. These machines are, due to manufacturing variation on the CPU, RAM, or whatever, much glitchier than the baseline. Once you've identified such a machine, you will want to replace it before it causes any persistent data corruption, not just transient crashes or glitches.

rygorous

2 ปีที่แล้วdiscuss

Person who actually did the investigation here. It took exactly one bug report.

RAD/Epic Games Tools is a small B2B company. Oodle has one person working full-time on it, namely me, and I do coding, build/release engineering, docs, tech support, the works. There's no multiple support tiers or anything like that, all issues go straight into my inbox. Oodle Data in particular is a lossless data compression API and many customers use two entry points total, "compress" and "decompress".

I get a single-digit number of support requests in any given month, most of which is actually covered in the docs and takes me all of 5 minutes to resolve to the customer's satisfaction. The 3-4 actual bug reports I get in any given year, I will investigate.

comments