Hitachi’s AD/ADAS combines the three elements of sense, think, and act, linking in-vehicle devices to the information infrastructure to provide functions that contribute to improving QoL through vehicles, including by enabling safe and comfortable driving by even elderly and beginner drivers. This article describes the next-generation stereo cameras to detect the various risks present in complex driving environments (sense) and the risk prediction map technologies based on the results of this sensing to predict and avoid collisions with surrounding objects (think). It also describes the vehicle control technologies used to operate chassis and power train actuators so that the driver can enjoy safe and comfortable mobility (act).
Hitachi Automotive Systems, Ltd. changed its name to Hitachi Astemo, Ltd. on January 1, 2021. The author’s position is as of December 2020.
Pursuing the goal of zero accidents, the development of automated driving (AD) and advanced driver assistance systems (ADAS) is progressing on a global scale, including in emerging nations. The resulting technology offers a way to turn vehicles into a means of improving quality of life (QoL) by making a major contribution to safe and comfortable mobility for beginner drivers and others who lack confidence in their driving as well as for the elderly who are concerned about their physical capabilities. Meanwhile, the impact of COVID-19 is expanding the scope of vehicle use from being a way to help people get around toward being a means of transporting goods due to greater logistics demand driven by personal consumption. This in turn has focused attention on how to provide last-mile delivery in ways that improve goods distribution efficiency. Meeting these societal needs requires the high-level integration of the three elements of sense, think, and act, which enable vehicles to detect their surroundings and move in a safe and efficient manner. In terms of improving QoL, the resulting technology is expected to help create a world in which everyone has the independence to travel freely by expanding the scope of this technology beyond highways and on to local roads, enabling these safe and comfortable vehicles to function in a wider range of driving situations.
This article describes the components of AD/ADAS, sensing technologies that extend their use to local roads, and vehicle control technologies that can transport people and goods in a safe and comfortable manner.
Figure 1 — Block Diagram of AD/ADAS
AD/ADAS is implemented through the coordinated operation of the vehicle’s sensors and ECUs.
While Japan’s population is rapidly aging, opinion polling on driving habits indicates that many elderly people regularly use vehicles as a means of transportation and that they drive themselves(1). Unfortunately, the changes in physical capability that come with age, which include a loss of dynamic visual acuity and reduced ability to make quick decisions, result in a tendency toward slower response times for steering and brakes. Nevertheless, many elderly people continue to operate vehicles despite concerns about their driving. The same is true of beginner drivers and others who lack confidence in their driving skills. Progress is being made on the development of ADAS and AD, two technologies seen as having the potential to overcome this issue.
Figure 1 shows the block diagram of an AD/ADAS. Vehicles with AD/ADAS are equipped with multiple sensors, including cameras and radar. Safe and comfortable driving requires 360° real-time detection of the region around the vehicle (sense). The AD/ADAS electronic control unit (ECU) integrates the information from these sensors to avoid collisions or other incidents by identifying and predicting the future location of all of the objects in the vicinity of the vehicle (other vehicles, people, and nearby objects such as traffic signals). It then utilizes the high definition map stored in the MAP ECU to plot a trajectory and velocity that will allow the vehicle to proceed safely, passing the resulting control instructions to the integrated control ECU (think).
The integrated control ECU handles control of the chassis (steering and braking, etc.) and power train (engine, etc.) systems, ensuring that the vehicle is controlled in accordance with the command inputs and what is happening around it, and in a way that combines safety with ride comfort (act). Objects and their predicted future locations identified by monitoring the region around the vehicle and the vehicle trajectory and velocity are forwarded to the Lumada’s Digital Innovation Platform in the form of driver records for collection and analysis, with this information then being passed on to various other vehicles for their collective use.
In this way, through the digitalization and accumulation of expert-level driving know-how, Hitachi’s AD/ADAS and its associated information infrastructure utilizes the Sense-Think-Act cycle to facilitate safe and comfortable driving even by beginner drivers, thereby helping to create a world in which everyone has the independence to travel freely.
The following sections describe stereo cameras with high-level object detection capabilities (sense), risk prediction maps used to predict and avoid potential collisions with these objects (think), and vehicle control technologies that ensure safety and ride comfort (act).
While past driver assistance systems have been intended for things like highway cruise control and lane keeping, such assistance will do more to improve QoL if it is targeted at driving on local roads. The problem is that situation assessment on local roads is much more difficult than it is on highways, with the region around the vehicle likely to contain a wider variety of other traffic and more complex road layouts and traffic rules to be followed. Obtaining an accurate picture of this complex environment requires three-dimensional sensing capable both of detecting specific objects and of operating at a distance over a wide area.
The following sections describe the use of a next-generation stereo camera to detect the various hazards present in complex traffic environments and the use of this detection information in risk prediction maps to predict and avoid potential collisions.
Hitachi has been supplying stereo cameras that simultaneously acquire images and three-dimensional information from a pair of cameras for use as driver assistance sensors since 2008. As shown in Figure 2, a stereo camera works like a pair of human eyes, using parallax (the difference between the images from the left and right cameras) to calculate the distance to an object. This parallax is transformed into a 3D image in which a distance value is associated with each point in the image, with all objects above a certain size that appear in this “parallax image” being tagged and recognition used to identify what they are. Among the features of Hitachi stereo cameras is their ability to detect objects of unknown shape or texture and to detect and range objects even if only partially visible. This makes them well suited to object detection in complex environments.
A Hitachi stereo camera that has been in commercial production since 2018 combines a horizontal angle of view for detecting pedestrians who step out in front of a moving vehicle with the long-distance vehicle detection capability needed for a car-following function called adaptive cruise control (ACC), two criteria used in the New Car Assessment Programme (NCAP) for evaluation of driver assistance functions.
Preventing accidents at intersections will be an important part of driver assistance on local roads in the future and NCAP is progressively incorporating this into its evaluations. Hitachi is currently developing a new next-generation stereo camera that combines the wide horizontal angle of view needed to prevent collisions with pedestrians or bicycles when turning at an intersection with the long-distance vehicle detection capability needed for the ACC car-following function(2).
As shown in Figure 3, the new camera expands the angle of view by around three times without significantly increasing the sensing pixel count by laterally offsetting the image angles of the left and right cameras. In addition to enlarging the region where the left and right camera angles overlap (where parallax can be used for object detection), the stereo camera also seamlessly combines this central region with the outer regions where the left and right camera fields do not overlap, which are obtained by processing sequential images from these regions with binocular vision. By doing so, it is able to provide precise detection and ranging of pedestrians or bicycles across the entire angle of view(3).
Figure 2 — Stereo Camera Distance Measurement Principles
The camera system uses parallax (the difference between the images from the left and right cameras) to generate a parallax image from which objects with specific shapes are identified.
Figure 3 — Next-generation Stereo Camera
While conventional method has been for both the left and right camera sensors to have the same angle of view (left), the new method features a wider angle of view for parallax object detection while also laterally offsetting the left and right camera sensors to achieve a wider overall detection spread (right).
Figure 4 — Risk Prediction Map Generation
The sensing results are used to predict potential collisions with objects on the road and these are plotted on a two-dimensional map of existing and potential risks. An obstacle avoidance trajectory is generated that avoids high-risk areas, with the velocity determined so as to allow for potential risks.
Vehicle sensors like the stereo camera described above are used to determine what is currently happening in the region around the vehicle so that visible hazards can be avoided. In complex environments like those found on local roads, however, there is a need to drive the vehicle in a way that allows for potential hazards hiding in places where sensing does not reach. Experienced drivers, for example, preemptively avoid hazards by unconsciously allowing for the possibility that a pedestrian might step out of the shadows. Likewise, driver assistance systems can assuage the concerns of many drivers and provide safety and comfort by predicting collision risks a few seconds in advance of detection and taking preemptive action to avoid them.
The risk prediction maps used for this purpose are generated by predicting the collision risks at locations that the vehicle has yet to reach and plotting the results of calculating these across the entire range of potential trajectories on a two-dimensional map (see Figure 4). These collision risks encompass both the existing risks of collision with static or moving objects detected by the sensors and the potential risks that may exist in areas the sensors cannot see (blindspots). In the case of existing risks, the system predicts the future path of moving objects and tags as high risk those locations where this path overlaps future vehicle positions. For potential risks, the system identifies blindspots created by the relative positions of the vehicle and detected objects and uses a predefined hazard model (such as the risk shown in Figure 4 of a pedestrian stepping out from the shadow of a parked car) to decide whether or not these constitute a potential risk. A forward-looking risk avoidance trajectory for the vehicle can then be obtained by overlaying these various risks on the risk prediction map and using it to find a route that stays within low-risk rather than high-risk areas. Likewise, preemptive risk avoidance can be achieved by varying the future vehicle velocity in such a way that, for example, the vehicle decelerates prior to reaching the location of risks such as places where a pedestrian might suddenly appear.
A proof-of-principle trial of the technique found that it should be possible to incorporate preemptive risk avoidance into automated driving with real-time risk prediction up to five seconds in advance.
Figure 5 — Realization Image for Dynamics Planning and HPVC
Mobility for enhanced QoL is achieved by a combination of Dynamics planning, HPVC, and the actuators that execute the resulting control actions.
Automated driving involves more than just getting from one place to another without having to drive the vehicle. Being able to travel safely and in comfort as well as quickly are important considerations for improving the QoL of users. Experienced drivers, for example, can drive in a way that does not discomfort their passengers even on winding roads. For automated driving and driver assistance to control vehicles in a way that improves QoL, they will need to implement this control on ECUs in such a way that the driving style satisfies the following three criteria.
Achieving these requires algorithms for calculating vehicle trajectories (route and velocity) that passengers find comfortable, vehicle control that can faithfully follow the calculated trajectory, and actuators to realize this vehicle control. Hitachi is developing a Dynamics planning algorithm for trajectory calculation and high-precision vehicle control (HPVC) for following this trajectory (see Figure 5).
Dynamics planning is a way of determining the safest and most comfortable vehicle trajectory taking account of the accelerations and jerks (changes in acceleration) experienced by the passengers, calculating it from the risk prediction map and the region of available trajectories in front of the vehicle. From the infinite number of route and velocity combinations possible through this available area, Dynamics planning calculates the smoothest trajectory that maintains ride comfort within the available road width. While it normally calculates a trajectory that includes sufficient early deceleration prior to entering sharp corners, the calculated trajectory also safely avoids situations where there are sudden changes in risk or corners that are too sharp to turn.
HPVC, meanwhile, provides the means to drive the vehicle around the sort of small corners that past driver assistance control technologies have found difficult to follow and to keep the vehicle accurately on track even when the trajectory calls for sudden steering changes. Past practice has been to target vehicle control slightly ahead of the actual vehicle position in order to avoid the loss of accuracy or stability caused by delays in actuators or other aspects of vehicle performance. This led to problems with the vehicle taking shortcuts, being unable to maintain the precise trajectory. HPVC overcomes the delay problem by running a predictive simulation of vehicle behavior in the ECU and is able to follow the precise trajectory by targeting control at a point close to the actual vehicle position.
Dynamics planning and HPVC can make a significant contribution to improving QoL through their ability to control vehicles in a way that provides safety and comfort.
This article has described work by Hitachi on AD/ADAS and on sensors and vehicle control technologies used in its implementation.
Through a Sense-Think-Act cycle that extends beyond the vehicle itself to include external information infrastructure (Digital Innovation Platform), the technologies described here will help create a world in which everyone has the independence to travel freely, with beginner drivers and the elderly able to drive safely and comfortably. In the future, Hitachi intends to combine and consolidate its extensive and diverse technologies as it works with vehicle manufacturers to further improve QoL as it relates to vehicles.
