A Closer Inspection of Tesla’s Autopilot Safety Statistics
Source: Tesla, Inc.
The automotive industry is at the beginning of a grand experiment. If completely successful, humanity could be ushered into a new economy where driving is a hobby, only for sunny days along clear roads with a view. The struggles and tedium of the daily commute could be handled by autonomous vehicles, traffic accidents could fall to nil, passengers could focus on working and relaxing in their mobile offices, and the elderly, disabled, and blind could have considerable mobility and autonomy. If a complete failure, automobile companies would have invested billions of dollars in computer vision, sensors, and automated driving systems only to have no effect on or actually increase the number of traffic accidents and fatalities by introducing new risks. This would cause a public backlash, and requiring regulators to impose a slow, costly review process that slows the pace of innovation so that after an initial roll-out to a few hundred thousand vehicles, further roll-outs are halted. Then, autonomous vehicle technology may follow the same path as the U.S. nuclear power industry, which has stopped building new power plants since the Three Mile Island accident in 1979. Which scenario or whether something in between unfolds depends on good design, as well as careful understanding and communication of the safety of autonomous driving technology and the path from partially autonomous to fully autonomous vehicles. And understanding the safety of autonomous vehicles (AV) is a very thorny statistics problem.
Recent fatal and injury-producing crashes involving vehicles with Tesla’s Autopilot and Uber’s self-driving pilot have led to significant disagreement among experts, reporters, automakers, and regulators about safety statistics for partial autonomy technologies[1]. Tesla, in particular, has made recent headlines after two crashes and 1 fatality with its Autopilot-equipped partial autonomy vehicles in the past few months. Tesla claims that its technology is 3.7x safer than the existing U.S. vehicle fleet, stating a fatality rate of 1 death per 86 million miles for conventional vehicles versus 1 death per 320 million miles for Autopilot-equipped vehicles, but many experts question the methodology and data behind these statistics. In this article, I’ll review the data, methods, and the three main criticisms of Tesla’s methodology for conventional vehicle fatality rates, provide my best estimates, and make recommendations for regulators and automakers on the safety of autonomous vehicles. I don’t have access to data to verify the fatality rate for Tesla Autopilot-equipped vehicles but the company has promised to release public Autopilot safety data in future quarters.
- What’s an Autopilot mile?
One informal complaint I’ve heard among analysts is the question of which miles should be included as an ‘Autopilot mile’ in Tesla’s statistic of 1 fatality per 320 million miles. Some analysts argue that one should only compare miles driven in a vehicle with Autopilot engaged to manually-driven vehicle-miles to obtain a fatality rate. Instead, Tesla’s methodology includes all miles driven with an Autopilot-enabled vehicle, whether or not the functionality was engaged.
I agree with Tesla’s methodology on Autopilot mileage because the road conditions under which a partial autonomy system is rated for operation (highways, clear lane markings, etc) are systematically different from manually-driven miles. If one only used Autopilot-enabled miles in the fatality rate calculation, a comparable baseline of miles for a manual vehicle driven under similar road conditions would be difficult to obtain and there are already considerable gaps in the vehicle mileage data needed to compute good partial autonomy safety statistics (more below).
Because the characteristics of manually-driven miles in Autopilot-enabled vehicles are very different than the miles driven in a manually-driven vehicle — more curves, poor lane markings, rain or poor-visibility weather, etc. — it could be possible that crashes are more likely to occur when an Autopilot-enabled vehicle turned over operation to the driver, because road conditions were worse. If that hypothesis were true, these types of crashes should be included as an Autopilot crash, as it pertains to the road coverage of Autopilot and the hand-off between autonomous and manual control, which is related to Tesla’s design choices.
So, unless the owner of an Autopilot-enabled vehicle never or rarely chose to enable the functionality, the proper comparison for fatality rate safety statistics should be made between Autopilot vehicles and all other vehicles.
- What are comparable vehicles and fatalities?
Another criticism of Tesla’s Autopilot safety statistics is aimed at its choice of comparable baseline vehicles in the 1 fatality per 86 million miles statistic. Analysts believe this statistic was obtained from the Insurance Insti