It's an exciting time for MicroVision, with the recent acquisition of Ibeo Automotive Systems GmbH representing a pivotal move toward its future.
Integrating Ibeo's perception software into MicroVision's lidar sensor was a key driver behind MicroVision's acquisition. CEO Sumit Sharma joins us to discuss perception software's benefits to automotive lidar systems and what the integration of perception software means for MicroVision's MAVIN™.
What is Perception in Autonomous Driving?
The environment around a vehicle is complex. When we drive, our eyes take in so much—the vast amount of information that comes with just a single object is immense. For instance: What kind of object is it? Is it a truck, a car, or a pedestrian? What are its dimensions? What is its position, and how far away from it are we? How fast is it accelerating? Is it braking?
Using all that information for every object we see while driving would overwhelm us—it is too much data. So, what we, as human drivers, do, is make the information abstract and simplified. "There is a truck coming in, and it is accelerating, so we need to slow down." This abstraction is precisely what the perception software does—it takes a massive amount of data and measurements and simplifies it, so when the autonomous car drives, the information is easy to process and can be used for decision-making.
When you think of the lidar system that MicroVision builds for driving functions, with the sensor and the perception software, you can think of it like a human driver. The sensor is like the eyes; it enables the vehicle to "see" its surroundings, collecting data from the environment to generate a point cloud. Perception software is like the brain, which interprets the visual data collected from the lidar sensor and turns it into useful information.
Perception’s Role in Automotive Lidar Systems
Perception software helps interpret complex data into information that advanced driver assistance systems (ADAS) or autonomous driving (AD) systems can understand so that the vehicle can react accordingly. The measurement data of the point cloud is fed into the perception software to produce an abstract representation of the environment.
Perception is one layer in the process of safe driving and is needed very early in the processing toolchain to provide fully autonomous driving:
- Sensing hardware to generate environment measurements such as point cloud, images, and vehicle speed.
- Perception to model the environment using environmental measurements. As a human, you might see something from the point cloud. But what the vehicle needs to know is, “That is another car, that is a pedestrian, there is where it can drive, there is where it should not go.” That’s an abstraction in the processing toolchain.
- Situation assessment uses perception to extract situation information. The vehicle assesses the situation and determines where it is. Is it on a highway, in a lane, or in traffic? Another example of situation assessment is when an ambulance is in the vicinity, as this situation holds another set of rules for necessary planning.
- Planning stage decides the vehicle's next steps to act using the identified rule set from the situation assessment. For example, "I am on the highway, so a lane change is allowed." Perception tells me there is a decelerating car in front, and if I want to maintain speed, I need to plan a lane change maneuver.
- Vehicle control is where there is actual action. Planning decides to brake and provides a planning trajectory to vehicle control. Vehicle control adjusts vehicle dynamics to match the stated planning trajectory.
Object Recognition and Lane Detection in Perception Software
One of the most important elements of the perception software process is object recognition—the ability to quickly identify and classify things on the road like pedestrians, cyclists, traffic signals, or other vehicles—and determine their location relative to the ego vehicle (the vehicle that contains the lidar sensor) and their velocity.
Object recognition is achieved using sophisticated algorithms to interpret complex visual information from cameras, radar sensors, and (most importantly) lidar. For scene interpretation in lidar, there are three general processing steps.
- Preprocessing and Clustering where point cloud segments belonging to one object are estimated.
- Detection and Model Fitting where point cloud segments are analyzed and characterized.
- Tracking and State Estimation, where analyzed point cloud segments are accumulated over time to estimate more information from just the distance, such as velocity or acceleration.
After interpreting what objects are in the environment, the car’s driving function makes decisions based on these interpretations.
Lane detection —another crucial element in perception software—quickly and accurately finds and tracks the lane markings in real-time, even in challenging conditions such as glare, insufficient lighting, or complex road configurations. Lane detection uses the same general three-step approach as described for object detection. Lane detection is essential to ADAS and AD as it provides information about the road layout, which is necessary for the situation assessment, and the ego vehicle's position within a lane, which is vital for safe navigation. Perception software aims to provide a comprehensive and accurate environment model for the driving function in real-time, including possible hazards before they pose a potential safety risk. Situation assessments can then report hazards using perception.
The Object List and Why It’s Important to OEMs
When the perception software first detects objects within its field of view, it uses algorithms to describe a rectangular bounding box around each object based on size and shape. The bounding box is the visualization of the object recognition result.
The result of object recognition is a list of objects where bounding box and extended information are stored. This object list represents one important perception interface to the OEM.
The object list contains information about each detected object in the environment. This includes details such as class (i.e., pedestrian, car, truck, motorcycle, bicycle, or obstacle) and, as shown in the image above, position, dimensions, and velocity. To estimate all these object details, point cloud data is analyzed to update the object state filter algorithm. These filter algorithms can identify motion patterns and accurately estimate object details if applied over time.
The bounding box allows the software to differentiate between one object from another and ensure that it is correctly identified. The box is a reference model for the object's location relative to the ego vehicle. If you were using perception software to detect a truck and a motorcycle in front of you on a highway, the system would draw a bounding box around each vehicle to track their movements in real-time and create an accurate model of the environment. The algorithms that make the driving decisions operate from the object list, which is why the object list is the most asked interface by customers.
Ibeo’s Perception Software Integrated into MAVIN™
MicroVision integrates the perception software stack developed by the Ibeo team into MAVIN's custom Application Specific Integrated Circuit (ASIC), which controls the sensor and processes data from it. Perception software was developed by Ibeo for over 20 years and has a proven track record. The integration enables MAVIN to collect data points and provide high-level information about the vehicle’s environment directly from the sensor.
What puts us in the pole position is that MicroVision provides both the sensor and the perception software from the same company. You can deliver the best product when the people creating the point cloud and the people using the point cloud in the perception are in one company. There are no limits in information exchange, no barriers between us.
The integration of perception in MAVIN provides OEMs with significant advantages. It reduces power consumption because ASIC is more efficient, lowers cost as external ECU hardware is expensive, and enables the OEM to focus on the driving function.
What MAVIN™ Perception Software Delivers to OEMs
The customer doesn't want to deal with complex environments—they need a simplified representation of the driving environment to put the driving function on top of. MicroVision's perception software provides customers with all the necessary information for ADAS and AD functionalities from Level 2+ to Level 5. This is terrific for OEMs because it saves them from having to develop the perception software on their own. Also, our customers are not burdened with considerable validation efforts because MAVIN delivers already-validated perception software.
Even if other lidar companies become adept at automotive software development and develop perception software relatively quickly, fulfilling all the automotive standards, including a long-term validation strategy, requires much work. MicroVision already meets the safety requirement standards.
OEMs who already have perception software and are interested in working with MAVIN's point cloud only can still utilize our perception software's output as a redundant path, a backup necessary for many safety-critical applications. OEMs’ safety concepts usually rely on a second path of information, checking that their first path doesn't contain an error. MAVIN can offer that right out of the sensor, even for customers who prefer the point cloud interface.
MAVIN™ offers perception interfaces like the object list and lane markings, as mentioned earlier. But many other interfaces—such as free space, small obstacle detection, vehicle positioning, and road boundaries—are also available. The free space interface, also known as the drivable space interface, details an area around the car that the car can safely use to maneuver. The small obstacle interface offers information about objects on the road, like tire parts or other debris, offering position and size, so the car can decide to go around or drive over it.
Having perception software for a lidar system makes it more reliable as it can detect obstacles in its environment with incredible accuracy—even at night or in bad weather conditions—and make smarter decisions about responding in any given situation.
And this is our goal at MicroVision: making the roads safer for everyone who uses them.