In his keynote at GTC, NVIDIA founder and CEO Jensen Huang announced NVIDIA Omniverse Replicator, a ground truth synthetic data generation engine for training AI networks. In a demo, Huang showed the power of Omniverse Replicator in developing autonomous vehicles with DRIVE Sim.
DRIVE Sim
DRIVE Sim is a simulation tool built on Omniverse that takes advantage of the platform's many capabilities. The data generated by DRIVE Sim is used to train deep neural networks that form the perceptual systems in autonomous vehicles. For the NVIDIA DRIVE team, synthetic data is an effective and important part of the AV development process.
The deep neural networks that drive an autonomous vehicle's perception are made up of two parts: an algorithmic model and the data used to train that model. Engineers have spent a lot of time refining the algorithms. However, the data side of the equation is still underdeveloped because real-world data is incomplete, time-consuming and costly to collect.
This imbalance often leads to a stalemate in DNN development and hinders progress when the data does not meet the requirements of the model. By generating synthetic data, developers have more control over data development and can tailor it to the specific requirements of the model.
Real data, while an important component for AV training, testing, and validation, also presents a significant challenge. The data used to train these networks is collected from sensors in a fleet of vehicles during real-world trips. After collection, the data must be labeled with ground truth. Labeling is done by hand by thousands of labelers - a time-consuming and costly process that can also be inaccurate.
Supplementing real-world data collection with synthetic data removes these bottlenecks and enables engineers to take a data-driven approach to DNN development, significantly speeding up AV development and improving real-world results.
The deep neural networks that drive an autonomous vehicle's perception are made up of two parts: an algorithmic model and the data used to train that model. Engineers have spent a lot of time refining the algorithms. However, the data side of the equation is still underdeveloped because real-world data is incomplete, time-consuming and costly to collect.
This imbalance often leads to a stalemate in DNN development and hinders progress when the data does not meet the requirements of the model. By generating synthetic data, developers have more control over data development and can tailor it to the specific requirements of the model.
Real data, while an important component for AV training, testing, and validation, also presents a significant challenge. The data used to train these networks is collected from sensors in a fleet of vehicles during real-world trips. After collection, the data must be labeled with ground truth. Labeling is done by hand by thousands of labelers - a time-consuming and costly process that can also be inaccurate.
Supplementing real-world data collection with synthetic data removes these bottlenecks and enables engineers to take a data-driven approach to DNN development, significantly speeding up AV development and improving real-world results.
THE PROBLEM OF THE GAP BETWEEN SIMULATION AND THE REAL WORLD
Generating synthetic data is a well-known tool for AI training - as early as 2016, researchers experimented with video games like Grand Theft Auto to generate data.
Unlike video games, however, the quality of DNNs is heavily influenced by how well the data matches the real world - training on datasets that don't translate to the physical world can actually degrade a network's performance.
This discrepancy between simulation and reality manifests itself in two primary ways. An appearance gap corresponds to pixel-level differences between the simulated image and the real image caused by how the simulator generates the data. The renderer, sensor model, fidelity of 3D assets, and material properties can all contribute.
A content gap can be caused by the lack of diversity in the real content and by differences between the simulation and real contexts. These inconsistencies occur when the context of a scene does not match reality. For example, in the real world there are dirty roads, dented cars, and emergency vehicles on the side of the road that must be reproduced in the simulation. Another important factor is the behavior of actors, such as traffic and pedestrians - realistic interactions are key to realistic data output.
Unlike video games, however, the quality of DNNs is heavily influenced by how well the data matches the real world - training on datasets that don't translate to the physical world can actually degrade a network's performance.
This discrepancy between simulation and reality manifests itself in two primary ways. An appearance gap corresponds to pixel-level differences between the simulated image and the real image caused by how the simulator generates the data. The renderer, sensor model, fidelity of 3D assets, and material properties can all contribute.
A content gap can be caused by the lack of diversity in the real content and by differences between the simulation and real contexts. These inconsistencies occur when the context of a scene does not match reality. For example, in the real world there are dirty roads, dented cars, and emergency vehicles on the side of the road that must be reproduced in the simulation. Another important factor is the behavior of actors, such as traffic and pedestrians - realistic interactions are key to realistic data output.
OMNIVERSE REPLICATOR TO NARROW THE GAP BETWEEN SIMULATION AND REALITY
This is where Omniverse Replicator comes in. It is designed to close the gap between appearance and content. To reduce the discrepancy between appearance and content, DRIVE Sim uses Omniverse's RTX path-tracing renderer to generate physics-based sensor data for cameras, radars, lidars, and ultrasonic sensors. Real-world effects are captured in the sensor data, including phenomena such as LED flicker, motion blur, rolling shutter, lidar beam divergence, and Doppler effect. These details even include realistic vehicle dynamics, which is important because, for example, the motion of a vehicle during a lidar scan affects the resulting point cloud.
The other half of this sensing equation is the materials. The materials in DRIVE Sim are physically simulated to obtain accurate beam reflections. DRIVE Sim includes a built-in lidar material library and a soon-to-be-released radar and ultrasound material library.
The other half of this sensing equation is the materials. The materials in DRIVE Sim are physically simulated to obtain accurate beam reflections. DRIVE Sim includes a built-in lidar material library and a soon-to-be-released radar and ultrasound material library.
.png)
One of the key ways to close the content gap is to create more diverse content with the highest level of fidelity. DRIVE Sim leverages the power of Omniverse to connect with a variety of content creation tools. However, creating appropriate scenes also requires the right context.
Under the surface, Omniverse Replicator organizes data for rapid scene manipulation using a technique called domain randomization. DRIVE Sim includes tools for this process and scene construction that generate a large amount of diverse data while maintaining real-world context. Since Omniverse Replicator is also temporally accurate and deterministic, the data set can be created in a repeatable manner.
Under the surface, Omniverse Replicator organizes data for rapid scene manipulation using a technique called domain randomization. DRIVE Sim includes tools for this process and scene construction that generate a large amount of diverse data while maintaining real-world context. Since Omniverse Replicator is also temporally accurate and deterministic, the data set can be created in a repeatable manner.
REAL RESULTS
DRIVE Sim has already achieved significant results in accelerating perceptual development with synthetic data at NVIDIA.
One example is the transition to the latest NVIDIA DRIVE Hyperion sensor set. The NVIDIA DRIVE Hyperion 8 platform includes sensors for full AV development in production. Even before these sensors were available, the NVIDIA DRIVE team was able to create DNNs for the platform using synthetic data. DRIVE Sim generated millions of images and real data for training. As a result, the networks were ready to go as soon as the sensors were installed, saving valuable months of development time.
In another case, the PathNet DNN, which detects drivable lanes, had difficulty determining a path when the vehicle was not in the center of the lane. Capturing such data is difficult because partially leaving the lane is dangerous (and violates NVIDIA's data collection policies). By training the network with millions of synthetic images of off-center lanes, DRIVE Sim was able to significantly improve PathNet's accuracy.
The same is true for LightNet, which detects traffic lights, and SignNet, which detects and classifies road signs. These networks had difficulty detecting traffic lights at extreme angles and misclassified signs under certain conditions because they lacked data. Engineers were able to develop data to augment the real-world datasets and improve performance.
By training both DNNs with synthetic data that covered these problem areas, performance improved quickly and bottlenecks in the development process were eliminated.
One example is the transition to the latest NVIDIA DRIVE Hyperion sensor set. The NVIDIA DRIVE Hyperion 8 platform includes sensors for full AV development in production. Even before these sensors were available, the NVIDIA DRIVE team was able to create DNNs for the platform using synthetic data. DRIVE Sim generated millions of images and real data for training. As a result, the networks were ready to go as soon as the sensors were installed, saving valuable months of development time.
In another case, the PathNet DNN, which detects drivable lanes, had difficulty determining a path when the vehicle was not in the center of the lane. Capturing such data is difficult because partially leaving the lane is dangerous (and violates NVIDIA's data collection policies). By training the network with millions of synthetic images of off-center lanes, DRIVE Sim was able to significantly improve PathNet's accuracy.
The same is true for LightNet, which detects traffic lights, and SignNet, which detects and classifies road signs. These networks had difficulty detecting traffic lights at extreme angles and misclassified signs under certain conditions because they lacked data. Engineers were able to develop data to augment the real-world datasets and improve performance.
By training both DNNs with synthetic data that covered these problem areas, performance improved quickly and bottlenecks in the development process were eliminated.
SEEING WHAT MAN CANNOT SEE
Synthetic data is changing the nature of DNN development. They are time and cost efficient and give engineers the freedom to create a customized dataset on demand.
Developers can specify elements such as weather, lighting, pedestrians, road pollution, and more. They can also control the distribution of elements, such as specifying a particular mix of trucks, buses, cars and motorcycles in a given data set.
Synthetic data provides ground truth that humans cannot name. Examples include depth information, speed, and multisensor tracking. This ground truth information can greatly enhance perceptual capabilities.
They also facilitate the labeling of components that are difficult or impossible to determine. For example, a pedestrian walking behind a car cannot be properly detected by a human if it is obscured. With simulation, however, ground truth is automatically available and accurate at the pixel level, even if the information is not visible to humans.
Developers can specify elements such as weather, lighting, pedestrians, road pollution, and more. They can also control the distribution of elements, such as specifying a particular mix of trucks, buses, cars and motorcycles in a given data set.
Synthetic data provides ground truth that humans cannot name. Examples include depth information, speed, and multisensor tracking. This ground truth information can greatly enhance perceptual capabilities.
They also facilitate the labeling of components that are difficult or impossible to determine. For example, a pedestrian walking behind a car cannot be properly detected by a human if it is obscured. With simulation, however, ground truth is automatically available and accurate at the pixel level, even if the information is not visible to humans.
CLEARING THE WAY FOR THE FUTURE
As a modular, open and extensible synthetic data generation platform, Omniverse Replicator provides deep learning engineers with powerful new capabilities. DRIVE Sim leverages these new features to give AV developers the ultimate flexibility and efficiency in simulation testing.
It enables engineers to create the data sets they need to accelerate their work.
The result is DNNs that are more accurate and can be developed in less time, bringing technology for safe and efficient autonomous driving to the road faster.
It enables engineers to create the data sets they need to accelerate their work.
The result is DNNs that are more accurate and can be developed in less time, bringing technology for safe and efficient autonomous driving to the road faster.