Have you ever wondered about the mystery surrounding the notorious Xbox 360 hardware failures? Why of course you have, who hasn’t? As you probably already know, Microsoft hasn’t been too vocal about the whole thing, always dodging the RRoD-related questions like bullets. However, if the company as a whole is tight-lipped about the issue, it doesn’t mean there aren’t individuals who feel like letting the cat out of the bag, as recently revealed by 8Bit Joystick.com and their exclusive interview with a Microsoft insider who’s worked on the console for many years and felt like it’s time to fill the public in on a number of burning issues.

First of all, it should be noted that 8Bit Joystick’s Jake was the one who broke the story about Bungie parting ways with Microsoft a full week before the official PR announcement, so if any of you have any doubts about his confidential source from inside Redmond, we suggest you let it go and prick your ears up. Now, let’s start with the question as to what led Microsoft to the situation it now finds itself in:

Q: Of all five videogame systems on the market now (PS3, PSP, PS2, DS, Wii and 360)only the Xbox 360 has had such major hardware failure problems. Microsoft being the only company based in the US making a videogame system. What part of Microsoft’s way of doing things do you think caused this situation to happen.
First, MS has under resourced that product unit in all engineering areas since the very beginning. Especially in engineering support functions like test, quality, manufacturing, and supplier management. There just weren’t enough people to do the job that needed to be done. The leadership in many of those areas was also lopsided in essential skills and experience. But I hear they are really trying to staff up now based on what has happened, and how cheap staff is compared to a couple of billion in cost of quality.

Second, MS was so focused on beating Sony this cycle that the 360 was rushed to market when all indications were that it had serious flaws. The design qual testing was insufficient and incomplete when the product was released to production. The manufacturing test equipment had major gaps in test coverage and wasn’t reliable or repeatable. Manufacturing processes at eall levels of suppliers were immature and not in control. Initial end to end yields were in the mid 30%. Low yields always indicate serious design and manufacturing defects. Management chose to continue to ship anyways, and keep the lines running while trying to solve problems and bring the yields up. Whenever something failed and there was a question about whether the test result was false, they would remove that test, retest and ship, or see if the unit would boot a game and run briefly and then ship. 360 is too complex of a machine to get away with that.

In the end I think it was fear of failure, ambition to beat Sony, and the arrogance that they could figure anything out, that led to the decision to keep shipping. That management team had made some pretty bad decisions in the past and had never had to pay a proportional consequence. I’m sure they thought that somehow they would figure it out and everything would end up ok. Plus, they tend to make big decisions like that in terms of dollars. They would rationalize that if the first few million boxes had a high failure rate, a few 10’s of millions of dollars would cover it. And contrasting that cost with a big lead on Sony, would pay it in a heartbeat. They weren’t even thinking about Nintendo.

Compare that to Sony, who delayed their launch, even though they were behind, when their box wasn’t ready.

With that said, the insider goes on to provide his opinion on what the main cause of the “Red Ring of Death” could be, as well as explains how games can cause hardware failures:

Q: In your opinion what do you think the main cause of the Red Ring of Death failures have been?
RROD is caused by anything that fails in the “digital backbone” on the mother board. Also known as a core digital error. CPU, GPU, memory, etc. Bad parts, incompatible parts (timing problems) bad manufacturing process (like solder joints), misapplied heat sinks or thermal interface material, missing parts, broken parts, parts of the wrong value, missed test coverage. Any one or more, on any chip, or many other discrete components, would cause this. And many of the failures were obviously infant mortality, where they work when they leave the factory and fail early in use. The main design flaw was the excessive heat on the GPU warping the mother board around it. This would stress the solder joints on the GPU and any bad joints would then fail in early life.

There are also other significantly high failure rates in other areas, like the DVD.

Q: Does some games more than others cause hardware failure. Gears of War and Dead Rising were thought to be system killers when they came out.
Of course. Infant mortality, which is a weakened mechanical “thing” like a solder joint with a void in it, are exercised to failure by cyclic stress. The number of cycles and the amplitude of temperature change from low to high determine how quickly it will fail. Certain games will consume more bandwidth on the GPU, which has the most substandard thermal solution on the mother board, making it a lot hotter, warping the mobo and flexing the solder joints. Weak joints fail quickly. The better the game, the more often it will be played, again accelerating failures.

In order to get more information about the RRoD failure and the things causing it, 8Bit Joystick offers up a number of interesting theories about the flaw which the insider then discusses and gives the verdict based on how close they are to the truth:

Q: Let’s go over some of the rumored reasons RROD. Could you tell how close each theory is?

Over heating CPU/GPU due to the lead free solder?
They don’t overheat due to PB Free. They over heat due to too much power dissipated in too small of an area, w/o a sufficient thermal management design to take the heat away from the junction of the transistors on the chips, the packages themselves, and the mobo. And the over heating is on the GPU. When the CPU heatsink is applied right, it does not over heat.

Defective parts due to overseas subcontractors?
Some defective parts, like BGAs where the solder balls are not of sufficient and uniform size, so they don’t solder down evenly, or the substrate is warped, causing some joints to have insufficient solder. Bad chips from marginal or under tested wafers. Others are deficient processes, like misaligning the solder paste to the circuit board, or same on the parts, or not having the thermal profile right in the reflow oven during soldering. Manufacturers new to PB free tend to err on the low temp side thinking they are saving the parts reliability wise from a large thermal load. What they are really doing is not reflowing the PB free solder enough to make a good joint. PB free solder is non eutectic, which means the different metals in the solder alloy melt at different temperatures, unlike leaded solder where everything melts at the same temperature. If you under heat it, it won’t bond well to the board or parts, won’t form a good joint, leaving voids and other defects in the joints that lead to early failure under normal circumstances. But when you add the extraordinary heat and mother board warpage that goes with it, well you get a catastrophic failure rate like we’ve all seen on 360.

Defective or insufficient heat sinks?
A heat sink like the one they eventually put on the GPU would have helped a lot, since it stops the GPU heat from warping the mobo and breaking the solder joints. The CPU heatsink was fine. I’ve heard the memory was running hot too, and contributing to these failures. Not sure if they were heated by contact with the GPU heatsink, proximity on the mother board, or both. But with the new GPU heatsink the failure rate probably would have still been double digits overall. Way too high still.

Corrupt BIOS or OS bricking the system?
Maybe. But haven’t heard of this outside of the periodic dash updates bricking boxes.

Is humidity a factor? Are Xbox 360s in Florida just as likely as a 360 in Seattle?
Humidity is a co-factor with temperature for many failure modes. The hotter the room ambient conditions, the more likely a 360 is to fail, all else being equal. Same for humidity.

Is keeping the 360 horizontal more safe than keeping it vertical?
I don’t think so. Vertical exposes more surface area and volume to heat exchange with cooler room air. And I think opens more vent holes. Just don’t let it fall over.

System wide design problems due to a production schedule that shipped a full year before the competition’s systems?
Yes. It just wasn’t mature enough. Too many design defects, lack of design margins, immature test processes and equipment, insufficient PB free manufacturing expertise at partner manufacturers who made the mother board.

While we’ve known about some of the rumoured causes for quite some time now, there are certainly more than a few bits of shocking information hiding in there that are going to make you think of the issue in a different way. For the rest of the interview, which goes on to topics such as the possibility of a redesigned 360 and the business aspect of Microsoft’s Entertainment & Devices Division, go here.

There must be a lot going through your mind after reading the highlights from the interview. Why not share your thoughts below?