Home

The Register has an article  on the findings of the NTSB investigation into the death of Elaine  Herzberg who was killed by an Uber test driver  who was more interested in looking at her phone than watching the road and being prepared to take control if the AI made a mistake.

https://www.theregister.co.uk/2019/11/06/uber_self_driving_car_death/

 

81 comments

Avatar
CygnusX1 [1208 posts] 1 week ago
0 likes

The forum topic is the headline from the article. I've fixed it below:

Remember the woman (supposedly in contol of the Uber self-driving car) that killed another woman who was crossing the street?

 

Avatar
hawkinspeter [4270 posts] 1 week ago
0 likes

You've got some trailing characters in your link. Try this instead: https://www.theregister.co.uk/2019/11/06/uber_self_driving_car_death/

Thanks for posting that, anyhow.

Avatar
CygnusX1 [1208 posts] 1 week ago
0 likes
hawkinspeter wrote:

You've got some trailing characters in your link. Try this instead: https://www.theregister.co.uk/2019/11/06/uber_self_driving_car_death/

Thanks for posting that, anyhow.

Yeah just noticed - and fixed.

Avatar
ktache [2228 posts] 1 week ago
0 likes

Cygnus, that link doesn't work for me, but here is a story from the LA times about how the vehicle wasn't programmed to spot pedestrians

https://www.latimes.com/business/story/2019-11-05/self-driving-uber-in-c...

Avatar
CygnusX1 [1208 posts] 1 week ago
0 likes

Here's a link direct to the full PDF report:

https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf

Avatar
CygnusX1 [1208 posts] 1 week ago
3 likes
ktache wrote:

Cygnus, that link doesn't work for me, but here is a story from the LA times about how the vehicle wasn't programmed to spot pedestrians

https://www.latimes.com/business/story/2019-11-05/self-driving-uber-in-c...

Link in original post should work now, I was fixing it whilst ol' Squirrel Nutkins (aka HawkinsPeter) was pointing it out as broken.  

 

But basically, yes, unless you used a designated crossing point, Uber's AI did not see you as a pedestrian. So jaywalkers were not seen as human (you could argue this may reflect the general thought process  of drivers  generally - AI or otherwise). 

Avatar
FluffyKittenofT... [2813 posts] 1 week ago
2 likes

The account talks repeatedly of 'jaywalking'.  Jaywalking is only a 'thing' in certain legal systems (notably the US).

 

  It's worrying if that concept, that is irrelevant to the UK, where there is no such thing, was built-in to the logic of the vehicle.  It all fits precisely what worries me about autonomous vehicles - that the law will changed to accomodate them at the expense of pedestrians and cyclists, rather than the reverse.

Avatar
hawkinspeter [4270 posts] 1 week ago
3 likes
FluffyKittenofTindalos wrote:

The account talks repeatedly of 'jaywalking'.  Jaywalking is only a 'thing' in certain legal systems (notably the US).

 

  It's worrying if that concept, that is irrelevant to the UK, where there is no such thing, was built-in to the logic of the vehicle.  It all fits precisely what worries me about autonomous vehicles - that the law will changed to accomodate them at the expense of pedestrians and cyclists, rather than the reverse.

I think that is more of a symptom of Uber being rubbish and cutting corners. It does seem that autonomous vehicles are a lot harder to get right than originally thought, but I still think that they have tremendous advantages in the long run. Give it another 10 years maybe.

Avatar
Sriracha [335 posts] 1 week ago
3 likes
FluffyKittenofTindalos wrote:

The account talks repeatedly of 'jaywalking'.  Jaywalking is only a 'thing' in certain legal systems (notably the US).

 

  It's worrying if that concept, that is irrelevant to the UK, where there is no such thing, was built-in to the logic of the vehicle.  It all fits precisely what worries me about autonomous vehicles - that the law will changed to accomodate them at the expense of pedestrians and cyclists, rather than the reverse.

Jaywalking does seem to have a particular hold over the American psyche. Just about every US TV prog or movie, there is a scene where the protagonists chase each other across an otherwise deserted street which seethes with traffic the moment they race across, barrel rolling over bonnets (hoods?) etc. Miraculously no one is run over, although the baddie escapes to prolong the drama when the goodie is encumbered by one final vehicle. By the next scene the traffic has melted away. Only in America.

Avatar
CyclingInBeastMode [155 posts] 1 week ago
0 likes

No surprise then that the US bods want cycle helmets and that 26,000 road deaths (which has been rising for a few years now) is seemingly acceptable collateral damage.

Avatar
Simon E [3866 posts] 1 week ago
2 likes

Christian Wolmar wrote a piece for CUK in last month. A snippet:

"my view is that the driverless car dystopia will never happen. Sure, there may be limited use for such vehicles, such as airport shuttles or other pre-fixed routes, but the vision of everyone being in driverless cars that are shared use is, frankly, a fantasy."

https://www.cyclinguk.org/cycle-magazine/christian-wolmar-are-driverless...

If you consider the complexity of programming something as complex as a car in a public space of ever-changing and unpredictable dimensions and hazards, factor in programmer error (like the Boeing 737 Max software) and the many possible points of failure, let alone the potentially disastrous chance of malware infection, it really does seem to be pie in the sky.

And at a time when there are more and more scientists, health professionals, NGOs and many others pushing for active travel, reduced numbers of cars in towns and cities (whatever their source of power), it has all the signs of an endless project that will suck up vast amounts of money and time but won't really produce satisfactory results.

Avatar
AlsoSomniloquism [211 posts] 1 week ago
1 like

I'm assuming the cameras they used for the footage might not have been high quality, but I'm pretty sure the woman pushing the bike was in total darkness until lit by the car lights a second before impact. Whilst it might have been avoided if the "vehicle operator" had been fully in control rather then being a bored passenger watching a phone or whatever. And definitely would have been avoided if the programmers had thought that roads might contain people anywhere, why was she crossing a road, in the dark, without waiting for the traffic to clear or sure they had slowed down? I suspect the accident would have happened if it was a normal car in those circumstances.

 

Avatar
hawkinspeter [4270 posts] 1 week ago
0 likes
AlsoSomniloquism wrote:

I'm assuming the cameras they used for the footage might not have been high quality, but I'm pretty sure the woman pushing the bike was in total darkness until lit by the car lights a second before impact. Whilst it might have been avoided if the "vehicle operator" had been fully in control rather then being a bored passenger watching a phone or whatever. And definitely would have been avoided if the programmers had thought that roads might contain people anywhere, why was she crossing a road, in the dark, without waiting for the traffic to clear or sure they had slowed down? I suspect the accident would have happened if it was a normal car in those circumstances.

If I recall, the video footage was much darker than you would expect from the general lighting conditions, so the pedestrian would have been much more visible than you'd expect from the car footage. Personally, I don't think that drivers can pay enough attention if they're not driving and are just being passengers until they're suddenly in the middle of a situation.

Avatar
Argus Tuft [108 posts] 1 week ago
0 likes

Two points-Why was the car driving at a speed inconsistent with it's headlight range?

 And don't these self driving cars have a radar system? Radar works in the dark doesn't it?

 

 

Avatar
ktache [2228 posts] 1 week ago
3 likes

It had both radar and lidar, both of which saw the pedestrian, but the software ignored her/believed her to be a stationary object.  The computer failed to track, or work out trajectory.

The Volvo's own pedestrian avoidance measures had been turned off by Uber as it resulted in a jerky driving style.

Christian Wolmar's blog makes for very good reading, his take on self driving cars is informative and refreshingly pessimistic.  His views on the railways is always knowledgable.

Avatar
hawkinspeter [4270 posts] 1 week ago
3 likes
Argus Tuft wrote:

Two points-Why was the car driving at a speed inconsistent with it's headlight range?

 And don't these self driving cars have a radar system? Radar works in the dark doesn't it?

The radar did pick up the pedestrian 5.6 seconds before the collision, but the AI didn't categorise correctly and predicted that she wasn't in the path of the car. Lidar then picked her up 5.2 seconds before, but again predicted no collision - it registered her as stationary.

Avatar
Philh68 [120 posts] 1 week ago
0 likes

Interesting report, deserves wider publicity. Converting to metric, 5.6 sec at 70kmh means the radar identified the victim 108 metres away. Average braking distance (without reaction time) on a dry road at that speed is 27 metres, so at that that point there was a considerable margin for safety.

What should have happened was a controlled braking to slow the vehicle, which would give the computers more time to ascertain the nature of the obstruction from the sensor input. Which is what a human would do, if you see something in your headlights but aren’t sure what it is or which way it’s moving.

The input-identify-react method used shows that engineers think of it as a mechanical problem, and don’t appreciate the subtlety and nuance of human decision making. Which tells me they’re a long way off making it work in all situations.

Avatar
Mungecrundle [1610 posts] 1 week ago
4 likes

The difference is that every day around the world hundreds if not thousands of pedestrians are involved in collisions with cars driven by humans. Whilst the individual driver may or may not learn something from the experience that does not filter out to all the other humans. With software, every single mistake, every unusual occurence, every unique scenario gets captured by a multitude of sensors, fed back for analysis and the lessons actually do get learned by all the linked AIs.

Can't be arsed to go find KSI figures for AI driven cars v human drivers but I seem to recall that already they are something like 1/10 as likely per mile to be involved in a RTC.

If nothing else, the sensors and software are already being installed into human driven vehicles in the form of the collision avoidance systems mentioned in the report that Uber engineers had turned off in the test car. Right now in the UK the only thing preventing autonomous vehicles driving certain stretches of motorway and major A roads is legislation.

Avatar
Argus Tuft [108 posts] 1 week ago
5 likes

The collision avoidance systems were turned off because they weren't working.After all the time and money that's been poured into autonomous cars we're nowhere near a vehicle that can travel on any road at any time under any conditions without human input.

And if a human has to be standing by ready to take over at any time he/she may as well be driving the bloody thing!

Avatar
Mungecrundle [1610 posts] 1 week ago
1 like
Argus Tuft wrote:

The collision avoidance systems were turned off because they weren't working.After all the time and money that's been poured into autonomous cars we're nowhere near a vehicle that can travel on any road at any time under any conditions without human input.

And if a human has to be standing by ready to take over at any time he/she may as well be driving the bloody thing!

Unrealistic expectation. No technology has ever been delivered in a perfectly safe form. Think of the history of commercial flight, or closer to home, the introduction of carbon fibre bicycle frames. Far fewer catastrophic failures these days.

My prediction is that some sections of British motorway will be opened to autonomous vehicles for large scale trials within 5 years.

Avatar
hawkinspeter [4270 posts] 1 week ago
2 likes

Just read a bit of ArsTechnica's report on this: https://arstechnica.com/cars/2019/11/how-terrible-software-design-decisions-led-to-ubers-deadly-2018-crash/

It sounds like Uber just kept on making bad design choices.

When the AI first saw the pedestrian, it thinks it sees an "other" object, the next moment it sees a stationary vehicle and each time it flips between "other" and "vehicle", it doesn't use the previous movement data and instead thinks that the pedestrian was stationary. Completely idiotic design.

Also, here's the info showing the difference between lighting conditions on the road in question and the Uber video: https://arstechnica.com/cars/2018/03/police-chief-said-uber-victim-came-from-the-shadows-dont-believe-it/

Avatar
Argus Tuft [108 posts] 1 week ago
1 like

i'm not talking about safety -it's capability that's the issue. Even now at this late stage of development these things can't deal with snow  and need  lane markings to operate. What percentage of the world's roads have lane markings?

Imagine you're driving in a rural town and you notice a massive hailstorm brewing up.A human might mount the kerb and shelter under a shop awning until it blows over. Would an autonomous vehicle? 5  years seems optimistic to me.

Avatar
Mungecrundle [1610 posts] 1 week ago
2 likes
Argus Tuft wrote:

i'm not talking about safety -it's capability that's the issue. Even now at this late stage of development these things can't deal with snow  and need  lane markings to operate. What percentage of the world's roads have lane markings?

Imagine you're driving in a rural town and you notice a massive hailstorm brewing up.A human might mount the kerb and shelter under a shop awning until it blows over. Would an autonomous vehicle? 5  years seems optimistic to me.

Unrealistic expectations. You seem to think that a complete and infallible technology is going to be delivered to the public at the first iteration. It isn't, no new technology ever is. Personally I'll be more than happy to drive my car to the A1, switch to autopilot for the journey north and be alerted to take back control whenever the system is less than 100% sure of its own decisions or reaches a section of the road system not designated as suitable for autonomous control.

As for capability. I suggest you book a Tesla road test. The system available today is more than capable of basic driving in traffic, but you have to keep your hands on the wheel for legal reasons.

With regards to snow, maybe you should check out some youtube videos (mostly featuring a boxy Lada for some reason) to see how shit humans are at driving in it.

Avatar
brooksby [5187 posts] 1 week ago
3 likes
Argus Tuft wrote:

And if a human has to be standing by ready to take over at any time he/she may as well be driving the bloody thing!

This!

At present, the machine learning systems don't seem to be up to everything they'll encounter in a typical urban (or rural, for that matter) road environment.

How many people are we willing to sacrifice so that the machines can learn, so that the later models are safer and can deal with the Real World?

Avatar
Simon E [3866 posts] 1 week ago
4 likes
Mungecrundle wrote:

With regards to snow, maybe you should check out some youtube videos (mostly featuring a boxy Lada for some reason) to see how shit humans are at driving in it.

We all know that you don't need snow to show shit driving. Here's one that occurred this morning:

https://twitter.com/samuriinbred/status/1192707129121165313

I'd prefer to see the money spent on:

- compulsory eye tests for all drivers and random roadside checks.

- driving retests every 5 years and required when 6 points accrued on licence.

- enforcement of speed limits and police to use dashcam/cycle cam footage. Increase the fines & sell impounded vehicles to pay for more police resources, both on the road and behind the desk.

- automatic ban on 12 points, no stupid excuses like 'financial hardship' (including for selfish gits like Alan Partridge). Suck it up, lowlife.

- caught driving when banned: vehicle crushed or impounded and sold, lifetime ban.

(I could probably think of a few more given a bit of time but aware that I'm straying a bit OT so I'll leave it at that).

Avatar
kil0ran [1733 posts] 1 week ago
1 like

All driving bans to carry a suspended prison sentence equivalent to the length of the ban. Caught driving whilst banned, banged up to enforce the ban.

Avatar
Argus Tuft [108 posts] 1 week ago
1 like
Mungecrundle][quote=Argus Tuft wrote:

i'm not talking about safety -it's capability that's the issue. Even now at this late stage of development these things can't deal with snow  and need  lane markings to operate. What percentage of the world's roads have lane markings?

Imagine you're driving in a rural town and you notice a massive hailstorm brewing up.A human might mount the kerb and shelter under a shop awning until it blows over. Would an autonomous vehicle? 5  years seems optimistic to me.

Unrealistic expectations. You seem to think that a complete and infallible technology is going to be delivered to the public at the first iteration. It isn't, no new technology ever is. Personally I'll be more than happy to drive my car to the A1, switch to autopilot for the journey north and be alerted to take back control whenever the system is less than 100% sure of its own decisions or reaches a section of the road system not designated as suitable for autonomous control. As for capability. I suggest you book a Tesla road test. The system available today is more than capable of basic driving in traffic, but you have to keep your hands on the wheel for legal reasons. With regards to snow, maybe you should check out some youtube videos (mostly featuring a boxy Lada for some reason) to see how shit humans are at driving in it.[/quote

We've different definitions of autonomy.What you're describing I'd define as glorified cruise control.

I'd only be interested in what we've been promised for so long-No controls or instruments,just a keyboard to type your destination into.

What's on offer now is pretty  lame.-You're sitting there with your hands on the wheel-but you're not "Driving"? And how much do you pay for that rewarding experience?

Avatar
ktache [2228 posts] 1 week ago
2 likes

Never happen kil0ran, I do like to watch Police Interceptors (catching wrong'uns, mostly in cars) and when they do catch banned or licenceless drivers they tend to get a paltry fine and a few extra points on their non existent driving licences.  Same with no insurance, the car is seized, but that just means a few extra quid in costs.  I do feel for these highly trained and proffessional Interceptors that their 'result' is often so derisory.

Avatar
Rich_cb [1041 posts] 1 week ago
4 likes

The Uber fatality was the result of the failure of two systems.

1. The AI system.
2. The human driver.

Using this example to demand the end to the development of system 1 in order to continue using system 2 indefinitely seems a bit short sighted.

The market leader in driverless cars has already launched a completely driverless taxi service in Phoenix, Arizona.

They have also clocked up 10 million driverless miles without, AFAIK, any deaths or serious injuries.

Avatar
AlsoSomniloquism [211 posts] 1 week ago
0 likes
hawkinspeter wrote:

Also, here's the info showing the difference between lighting conditions on the road in question and the Uber video: https://arstechnica.com/cars/2018/03/police-chief-said-uber-victim-came-from-the-shadows-dont-believe-it/

Unless the original Uber video was deliberately doctored, I would argue that the lights were off along that stretch on that day as only one in the distance at the junction is showing and none of the others that should have appeared are showing at all. I wonder if the NTSB full report will mention that or not. And as for the headlight distance, unfortunately we now have cars with adjustable dipping, supposedly to stop blinding of people if you put too much weight in the back to lift the front. Unfortunately most people set and forget so might have been on lowest setting. That woiuld have been on Uber / Vehicle operator.

However it still doesn't stop the fact that the ped / cyclist stepped in front of a car to cross a road with the car in view  (and probably computer controlled to do the legal speed limit on that stretch) and expected the driver to move out of her way by either braking or switching lanes.  

Pages