Toyota's accelerating issue

[Scott K]

I am not sure what the standard was back then, so, I am not sure if all manufacturers were lacking in their design for their software, or if it was unique to Toyota's? But it seems it wasnt all drivers error.

Link

Toyota's killer firmware: Bad design and its consequences

On Thursday October 24, 2013, an Oklahoma court ruled against Toyota in a case of unintended acceleration that lead to the death of one the occupants. Central to the trial was the Engine Control Module's (ECM) firmware.

Embedded software used to be low-level code we'd bang together using C or assembler. These days, even a relatively straightforward, albeit critical, task like throttle control is likely to use a sophisticated RTOS and tens of thousands of lines of code.

With all this sophistication, standards and practices for design, coding, and testing become paramount – especially when the function involved is safety-critical. Failure is not an option. It is something to be contained and benign.

So what happens when an automaker decides to wing it and play by their own rules? To disregard the rigorous standards, best practices, and checks and balances required of such software (and hardware) design? People are killed, reputations ruined, and billions of dollars are paid out. That's what happens. Here's the story of some software that arguably never should have been.

For the bulk of this research, EDN consulted Michael Barr, CTO and co-founder of Barr Group, an embedded systems consulting firm, last week. As a primary expert witness for the plaintiffs, the in-depth analysis conducted by Barr and his colleagues illuminates a shameful example of software design and development, and provides a cautionary tale to all involved in safety-critical development, whether that be for automotive, medical, aerospace, or anywhere else where failure is not tolerable. Barr is an experienced developer, consultant, former professor, editor, blogger, and author.

Barr's ultimate conclusions were that:

Toyota’s electronic throttle control system (ETCS) source code is of unreasonable quality.

Toyota’s source code is defective and contains bugs, including bugs that can cause unintended acceleration (UA).

Code-quality metrics predict presence of additional bugs.

Toyota’s fail safes are defective and inadequate (referring to them as a “house of cards” safety architecture).

Misbehaviors of Toyota’s ETCS are a cause of UA.

A darning summary to say the least. Let's look at what lead him to these conclusions:

Hardware

Although the investigation focused almost entirely on software, there is at least one HW factor: Toyota claimed the 2005 Camry's main CPU had error detecting and correcting (EDAC) RAM. It didn't. EDAC, or at least parity RAM, is relatively easy and low-cost insurance for safety-critical systems.

Other cases of throttle malfunction have been linked to tin whiskers in the accelerator pedal sensor. This does not seem to have been the case here.

The Camry ECM board. U2 is a NEC (now Renesas) V850 microcontroller.

Software

The ECM software formed the core of the technical investigation. What follows is a list of the key findings.

Mirroring (where key data is written to redundant variables) was not always done. This gains extra significance in light of …

Stack overflow. Toyota claimed only 41% of the allocated stack space was being used. Barr's investigation showed that 94% was closer to the truth. On top of that, stack-killing, MISRA-C rule-violating recursion was found in the code, and the CPU doesn't incorporate memory protection to guard against stack overflow.

Two key items were not mirrored: The RTOS' critical internal data structures; and—the most important bytes of all, the final result of all this firmware—the TargetThrottleAngle global variable.

Although Toyota had performed a stack analysis, Barr concluded the automaker had completely botched it. Toyota missed some of the calls made via pointer, missed stack usage by library and assembly functions (about 350 in total), and missed RTOS use during task switching. They also failed to perform run-time stack monitoring.

Toyota's ETCS used a version of OSEK, which is an automotive standard RTOS API. For some reason, though, the CPU vendor-supplied version was not certified compliant.

Unintentional RTOS task shutdown was heavily investigated as a potential source of the UA. As single bits in memory control each task, corruption due to HW or SW faults will suspend needed tasks or start unwanted ones. Vehicle tests confirmed that one particular dead task would result in loss of throttle control, and that the driver might have to fully remove their foot from the brake during an unintended acceleration event before being able to end the unwanted acceleration.

A litany of other faults were found in the code, including buffer overflow, unsafe casting, and race conditions between tasks.

[delcecchi]

But I bet the plantiff's expert witness didn't reproduce the error, as no one has been able to do so far as I know. The writeup found no smoking gun.

It does sound like the software was sort of crufty, but basically they blew a bunch of smoke up the jury's behind.

Perhaps they could explain why every test that I saw (didn't see them all) showed that with the pedal to the metal the car could still be stopped by the brakes.

Was the jury really capable of evaluating this type of evidence?

[Jeremy airjer W]

But I bet the plantiff's expert witness didn't reproduce the error, as no one has been able to do so far as I know. The writeup found no smoking gun.

It does sound like the software was sort of crufty, but basically they blew a bunch of smoke up the jury's behind.

Perhaps they could explain why every test that I saw (didn't see them all) showed that with the pedal to the metal the car could still be stopped by the brakes.

Was the jury really capable of evaluating this type of evidence?

Agreed!

[Black_Bay]

Perhaps they could explain why every test that I saw (didn't see them all) showed that with the pedal to the metal the car could still be stopped by the brakes.

It's very easy to stop a car with the brake with the throttle wide open when 1)The driver knows it's coming, 2) The driver has a clue what to do in the case of UA, and the biggest one 3)The person has a clue how to drive in the first place. It's similar to making OJ put on the glove.

[delcecchi]

It's very easy to stop a car with the brake with the throttle wide open when 1)The driver knows it's coming, 2) The driver has a clue what to do in the case of UA, and the biggest one 3)The person has a clue how to drive in the first place. It's similar to making OJ put on the glove.

What is so tricky about "car is accelerating, step on brake hard.". Maybe supplemented by "turn off key".

#darwin

[Whoaru99]

What is so tricky about "car is accelerating, step on brake hard.". Maybe supplemented by "turn off key".

#darwin

There is nothing tricky about it being fundamentally easy. The tricky part is human reaction in an unexpected panic situation.

[JOHNBIGDOG]

I thought this too. Untill I got a call from my mother one day.

She stated she was locked inside her truck and couldn't get out because the interior door handle broke off. ( gmc 1500wt, passenger sode of truck full of product for a craft sale ) All I could do was laugh out loud and calmly state, "roll down the window." She lost it right there. So after 5 mineuts of crying and yelling and asking me to come home to get her out, I calmly explained that the exterior door handle still works.

If my car went W.O. I think the first thing I would do is throw it in neutral. But hey I haven't had it happen yet. Maybe its the technician in me. I always think using the trans. Droping it between D. And R. is a great time.

[Whoaru99]

I always think using the trans. Droping it between D. And R. is a great time.

It is great fun, long as it lasts. Even the mighty TH400 eventually succumbs to the reverse-lows.

[Jeremy airjer W]

I thought this too. Untill I got a call from my mother one day.

She stated she was locked inside her truck and couldn't get out because the interior door handle broke off. ( gmc 1500wt, passenger sode of truck full of product for a craft sale ) All I could do was laugh out loud and calmly state, "roll down the window." She lost it right there. So after 5 mineuts of crying and yelling and asking me to come home to get her out, I calmly explained that the exterior door handle still works.

I got one better than that.

There was a person who made a phone call to explain that the battery went dead and the car stalled on the way home. That person was panicking because they could not get out of the car because the power locks wouldn't work along with the power windows. It was explained to that person that the ford Taurus that they where driving would unlock the door by simply pulling on the inside door handle. A face palm moment then followed.

We had a GS lock himself in a car one time. Took him a couple minutes to figure out how to push the unlock button.

I towed a customer one time that couldn't tell us where she was. She gave us landmarks and eventually I found her. Upon greeting her at the car I showed her the street sign she was parked under clearly stateing the two streets that she happened to break down at.

I can go on....

[delcecchi]

One of the scariest things I had happen was in my 76 Mercury Montego wagon (Moby Mesabi, becuz it was big a a whale and rapidly returning to iron oxide). It was a cold night and I was driving home from work on Highway52 when I stepped on the gas and it stayed stepped on due to frozen throttle cable. Shut off ignition, steering locks. shift to neutral, turn key back on to unlock steering. pull to side of road. clean out pants. wait a few minutes. restart car, drive home.

[BoxMN]

I had a Ford Ranger that the throttle cable would stick. Maybe moisture got into it, not sure, but happened on moist nights below freezing. I would then have to go hit the cable with heavy flashlight until it "unbound".

Also recently had my Malibu stick at about 2500 rpm, and holy cow did it take some braking to stop it. Can't recall what it was, some type of electronic sensor they fixed, and no probs since. It is unsettling to say the least.