Self-driving cars were supposed to be in our garages by now, according to the optimistic predictions of just a few years ago. But we may be nearing a few tipping points, with robotaxi adoption going up and consumers getting accustomed to more and more sophisticated driver-assistance systems in their vehicles. One company that’s pushing things forward is the Silicon Valley-based Helm.ai, which develops software for both driver-assistance systems and fully autonomous vehicles.
The company provides foundation models for the intent prediction and path planning that self-driving cars need on the road, and also uses generative AI to create synthetic training data that prepares vehicles for the many, many things that can go wrong out there. IEEE Spectrum spoke with Vladislav Voroninski, founder and CEO of Helm.ai, about the company’s creation of synthetic data to train and validate self-driving car systems.
How is Helm.ai using generative AI to help develop self-driving cars?
Vladislav Voroninski: We’re using generative AI for the purposes of simulation. So given a certain amount of real data that you’ve observed, can you simulate novel situations based on that data? You want to create data that is as realistic as possible while actually offering something new. We can create data from any camera or sensor to increase variety in those data sets and address the corner cases for training and validation.
I know you have VidGen to create video data and WorldGen to create other types of sensor data. Are different car companies still relying on different modalities?
Voroninski: There’s definitely interest in multiple modalities from our customers. Not everyone is just trying to do everything with vision only. Cameras are relatively cheap, while lidar systems are more expensive. But we can actually train simulators that take the camera data and simulate what the lidar output would have looked like. That can be a way to save on costs.
And even if it’s just video, there will be some cases that are incredibly rare or pretty much impossible to get or too dangerous to get while you’re doing real-time driving. And so we can use generative AI to create video data that is very, very high-quality and essentially indistinguishable from real data for those cases. That also is a way to save on data collection costs.
How do you create these unusual edge cases? Do you say, “Now put a kangaroo in the road, now put a zebra on the road”?
Voroninski: There’s a way to query these models to get them to produce unusual situations—it’s really just about incorporating ways to control the simulation models. That can be done with text or prompt images or various types of geometrical inputs. Those scenarios can be specified explicitly: If an automaker already has a laundry list of situations that they know can occur, they can query these foundation models to produce those situations. You can also do something even more scalable where there’s some process of exploration or randomization of what happens in the simulation, and that can be used to test your self-driving stack against various situations.
And one nice thing about video data, which is definitely still the dominant modality for self-driving, you can train on video data that is not just coming from driving. So when it comes to those rare object categories, you can actually find them in a lot of different data sets.
So if you have a video data set of animals in a zoo, is that going to help a driving system recognize the kangaroo in the road?
Voroninski: For sure, that kind of data can be used to train perception systems to understand those different object categories. And it can also be used to simulate sensor data that incorporates those objects into a driving scenario. I mean, similarly, very few humans have seen a kangaroo on a road in real life. Or even maybe in a video. But it’s easy enough to conjure up in your mind, right? And if you do see it, you’ll be able to understand it pretty quickly. What’s nice about generative AI is that if [the model] is exposed to different concepts in different scenarios, it can combine those concepts in novel situations. It can observe it in other situations and then bring that understanding to driving.
How do you do quality control for synthetic data? How do you assure your customers that it’s as good as the real thing?
Voroninski: There are metrics you can capture that assess numerically the similarity of real data to synthetic data. One example is you take a collection of real data and you take a collection of synthetic data that’s meant to emulate it. And you can fit a probability distribution to both. And then you can compare numerically the distance between those probability distributions.
Secondly, we can verify that the synthetic data is useful for solving certain problems. You can say, “We’re going to address this corner case. You can only use simulated data.” You can verify that using the simulated data actually does solve the problem and improve the accuracy on this task without ever training on real data.
Are there naysayers who say that synthetic data will never be good enough to train these systems and teach them everything they need to know?
Voroninski: The naysayers are typically not AI experts. If you look for where the puck is going, it’s pretty clear that simulation is going to have a huge impact on developing autonomous driving systems. Also, what’s good enough is a moving target, same as the definition of AI or AGI[ artificial general intelligence]. Certain developments are made, and then people get used to them, “Oh, that’s no longer interesting. It’s all about this next thing.” But I think it’s pretty clear that AI-based simulation will continue to improve. If you explicitly want an AI system to model something, there’s not a bottleneck at this point. And then it’s just a question of how well it generalizes.
From Your Site Articles
Related Articles Around the Web