Hcfyk

Please don't point out APUs with x86_64 cores used in current generation game consoles, these are not part of the question

I cannot recall a single arcade system or game console that ever used x86 for its CPU. I'm happy to be corrected in the comments if there were some. Notwithstanding these exceptions, which must be incredibly rare, it sure seems that gaming hardware steered away from this otherwise incredibly popular CPU family.

Why is this the case? What tradeoffs in game hardware design likely led engineers away from using x86 for the CPU?

Video game hardware, whether for home consoles or arcade machines, is designed pretty much from scratch. Hardware designers have pretty much free rein on choosing what CPU to use, basing their choice on factors like cost and ease of programming. The Intel 8086, quite frankly, was a poorly designed processor and was never well regarded. While you could argue it made reasonable compromises at the time it was released (1978), these compromises ended up hanging around its neck like an albatross. If IBM hadn't picked the Intel 8088 for its Personal Computer in 1981, you probably wouldn't be asking this question.

We take the x86 architecture for granted today, but before the IBM PC it was fairly obscure and afterwords widely ridiculed. In particular, it compared poorly with the Motorola 68000 which had a flat 24-bit address space, more orthogonal instruction set and sixteen 32-bit registers. The 8086 used a segmented 20-bit address space, placed more restrictions on how various registers could be used, and only had eight 16-bit registers. It also wasn't particularly cheap, though the 8088 with its 8-bit data bus helped reduce overall costs compared to the 8086.

During the 70s and first half of the 80s, 16-bit CPUs like the 8086 and 68000 weren't really much of a consideration. The games of this era didn't demand anything more powerful than an 8-bit Z80 or 6502. While there were Gottlieb/Mylstar arcade games like Q*Bert in the early 80s that used a 5 MHz 8088 CPU, it's not clear what advantage this gave the machines. Performance in games of this era was mostly limited by how fast the CPU could access memory. Because of how the 8086/8 was designed, this made the 8088 effectively about as fast as a Z80 or 6052. These Gottlieb/Mystar arcade games also only had 64k (16-bit) memory maps, so they didn't benefit from the 8088's 20-bit address space.

Starting around the mid-80s, games had started moving beyond the capabilities of 8-bit CPUs. While the dominance of the IBM PC in the personal computer market at this point would've meant there would be programmers out there familiar with the 8086, there would've been few people singing its praises. By and large 68000 CPUs were chosen for new arcade game hardware designs that needed more power than 8-bit CPUs offered. Console hardware, being more cost sensitive, stuck with 8-bit CPUs for the rest of the decade, though most of the next generation went with 16-bit CPUs, either the 68000 or 65816. It's also worth mentioning that the two major new home computer designs of the mid-80s, the Commodore Amiga and Atari ST, also went with the 68000.

While arguably the 80386, introduced in 1985, solved a lot of the 8086's problems, with a more orthogonal instruction set, 32-bit flat address space and 32-bit registers, it wasn't until the early 90s that games started demanding the level of performance it offered, and when its price would have dropped to make it competitive in new hardware designs. It's not entirely clear to me why it didn't attract more interest at this point. The early 90s was also about the time that the IBM PC became the premier platform for home computer gaming. It would've inherited the disdain its predecessors had, but there were some arcade boards designed in the early 90s that use the 8086-compatible NEC V30 type CPUs. I think the main factor against at the time was that back then RISC-based architectures were considered the future, while CISC-based architectures like the x86 and 68k were considered obsolete. Still, that didn't stop Sega from using the CISC-based NEC V60 CPU in its arcade hardware designs in the early 90s.

For the rest of the 90s though, RISC-based CPUs like the Hitachi SH and IBM PowerPC, dominated arcade hardware designs -- at least at the high performance end. At the lower performance end, cheaper 68k and NEC V30 based designs were still in use. In the home console market, the 5th generation was almost all RISC CPUs, though notably the Japan-only FM Towns Marty used an AMD 386SX CPU. For the most part, this situation continues to around the turn of the century, with both arcade games and the 6th generation of consoles.

A big exception is Microsoft's Xbox. A 6th generation console, released in 2001, it has an Intel Pentium III CPU, much like PCs of the time. It's not surprising that Microsoft, with its long experience using the x86 CPU, made this design choice, but it's only a few years after this that mainstream Intel and AMD CPUs start appearing in arcade hardware. Although these x86-based arcade machines aren't really new hardware designs, they're PC clones running Windows or Linux. The 7th generation of home consoles went exclusively with PowerPC CPUs, but I suspect this had more do with the prices IBM was offering rather than the relative technical merits of the CPUs. Arcade games went increasingly with PC clone based hardware.

Today the choice of CPU in current game hardware designs is unremarkable. Home consoles and arcade games use x86 CPUs just like our personal computers do. Handheld consoles use ARM CPUs just like our phones do.

So, in the early days of game hardware design, x86 CPUs weren't chosen simply because there wasn't good reason to use one except for IBM PC compatibility. Later 32-bit x86 CPUs solved a lot the architecture's problems but RISC CPUs were seen as more modern. Today the ubiquity x86 architecture combined with its unrivalled speed has turned it into the dominant CPU architecture for game hardware that doesn't need to run off a battery.

Konix Multisystem: 6 MHz 8086 (1989).

Sure, it was cancelled just before release, but it got amazing press (I remember Jeff Minter raving about it at Earls Court) and some of it lived on as the (68k based) Atari Jaguar.

The original 8086 was quickly overshadowed by the Z80, which was somewhat compatible but easier to work with as it required less support hardware. Also many arcade developers preferred the 6502 and derivatives, and then later the 68000 which was easier to work with on both the hardware and software fronts.

Another issue was that the development machines available for testing code were often 68000 based as well. One prime example was the Sharp X68000. A lot of game programmers of that era were self taught too, and for hobbyist home computer systems Z80 and 6502 dominated with very few using 8086.

Finally, the 8086 was much more expensive than the Z80, while offering no real advantages over it unless you were expecting to buy millions and sell a line of compatible computers for years to come.

The original Xbox was an ~$800 computer sold at a loss, with embedded hardware making it impossible to use it as such. To my knowledge, Microsoft was the first company to take that gamble: that it couldn't be hacked and used as a home PC, and therefore negate the sales of their peripherals or software that they get kickbacks on. They took that gamble because (they're a computer company, not a gaming company) they could afford it, and they won big because they had competent people design the system, and a marketing strategy to suit.

I have no info about the gaming consoles but when I was creating my own HW computing stuff (for different purposes) I did not use Intel CPUs as they require additional ICs just to be able to work (It stuck till today as the PCs have a chipsets on board...). Z80 and or MCUs (even from Intel) was preferred by me as it did not need those and you know fewer ICs is usually cheaper, easier to design and manufacture PCB ... Once MCU's computing power matched my needs I stick with those and using them till today in my designs.

So my bet is that in early day game console designers used similar reasoning ...