Skip to main content

Hitachi

Corporate InformationResearch & Development

Surveillance cameras installed along streets and in shops, etc. are playing an essential role in assuring the safety and security of societies. The powerful enemy of the surveillance camera, however, is fog—which makes images look hazy. Hitachi Ltd. has launched a surveillance camera featuring a function that makes subjects clearly visible even if they are shot on foggy streets. Engaged in development of surveillance cameras under the watchword "better vision than the human eye," our researchers spoke to us in the interviews transcribed below.

Challenges facing surveillance cameras

Surveillance cameras are being installed in various places around towns.

Photo: SAI Hirotomo

SAISurveillance cameras are installed in places like train stations and spaces housing bank ATMs. Triggered by the major terrorist attacks of recent years, safety awareness has been growing, and surveillance cameras have proliferated in towns and cities. Recently, it is becoming desirable to install surveillance cameras in buildings with the aim of confirming safety and conditions in the case of a fire in the manner of evacuation guidance such as "There is no fire this way."

In other situations, like retail stocktaking, education, and nursing care, surveillance cameras are actively taking the place of the human eye.

Surveillance cameras are actively replacing the human eye, aren't they.

Photo: HIROOKA Shinichiro

HIROOKAAlthough we tend to think surveillance cameras are replacing the human eye, in fact, there are certain situations in which they cannot do so. For example, try to imagine a surveillance camera installed at entrances of places like apartment buildings. In most cases, the camera is set up so it points from the inside to the outside. When the camera is set up in this manner, since the outside is usually bright, when a person enters the entrance, the backlighting due to the bright outside causes the face of that person to be totally underexposed (i.e., "blacked-out"). If we looked with our own eyes, we would see the face properly; however, the fact of the matter is the camera cannot "see" the face properly. In the case of home-use video cameras, the subject is captured while the camera operator is looking at it, so if the operator notices backlighting, he or she can take action to avoid it. That cannot be said of a surveillance camera, however. A surveillance camera is required to shoot properly in any scene.

Photo: USUI Tsutomu

USUIAt present, backlighting is countered with a technique called "contrast enhancement"—which improves captured video images by processing dark portions and light portions. Contrast enhancement is a technique that determines and corrects bright portions and dark portions of images captured by the camera in a manner similar to the way in which the human eye judges brightness. By means of this technique, it is possible to recognize a person's face even if the image is captured in the presence of backlight and to capture images without overexposing excessively bright portions (i.e., "whiteout").

However, other than backlighting, other challenges face surveillance cameras installed outdoors—namely, the majority of cases. One such challenge is taking countermeasures against fog.

Figure 1: Contrast-enhancement technology

Assessing the denseness of fog and reducing it

When fog is present, subjects become blurred and images cannot be captured.

HIROOKAThat's right. From the viewpoint of a surveillance camera installed outside, fog is the powerful enemy. When fog appears, captured video images become faded. This fading causes problems such as the impossibility of determining whether people are passing by and the difficulty in making out car number plates. Given the need to fix these problems somehow, we started research on a technology to reduce the fog ("defogging" hereafter) from images captured by outdoor surveillance cameras.

USUISince contrast-enhancement technology was already available, we decided to apply it to defogging. Fading of images captured in foggy conditions is due to the signal forming the image being "squeezed," so to speak; that is, the image contrast is reduced. Even so, if we look at actual data, we see that a small "squeezed" signal remains. So we thought of reducing the fog by utilizing that signal.

How do you reduce the fog?

HIROOKAThe remaining signal is "expanded" in accordance with the denseness of the fog. When fog is present, it is not uniformly dispersed, so within a captured image, some portions contain dense fog, and other portions contain sparse fog. Accordingly, each portion of an image is analyzed to determine how much fog it contains. The amount that contrast must be corrected is determined on the basis of the results of that analysis; that is, portions containing dense fog have their contrast adjusted accordingly, and the portions containing sparse fog are processed to an eye-friendly extent. After that, by considering atmospheric scattering, in what way brightness should be constricted becomes clear, so the condition before the fog appeared is reproduced on the basis of the degree of constriction.

Figure 2: Flow of the defogging process

Figure 3: Images before and after defogging

USUIIn a surveillance camera, various components—such as sensors for converting captured images to signals, a CPU for controlling the signal processing, and hardware for performing image processing—are mounted. As for the defogging process, the CPU evaluates the fog condition, and the right amount of correction is determined. On the basis of the evaluation result, the hardware executes the image processing to achieve defogging.

The image processing, as well as the determination of fog denseness, is done entirely on the camera, isn't it?

SAIThat's right. The latest surveillance cameras transmit images to central control systems and recorders via LANs and so on. To suppress the data volume, the data is compressed during transmission. As a result of this data compression, lack of information occurs. As I explained in regards to there being a remaining signal on close inspection, effectively reducing the fog necessitates a complete set of data. That is to say, it is necessary to perform the defogging on the camera side before the signal is transmitted.

Evaluation by experiment and human eye

How do you determine whether defogging has been successful or not?

Photo: HIROOKA Shinichiro

HIROOKAQuantitative evaluation by experiment and subjective evaluation by human eye are performed.

In the experiments, by setting up an environment that generates fog indoors, we repeatedly measured the extent to which the images are improved while applying the defogging technology. There is an index called "visibility in meters" to describe the situation that fog is present. However, this index varies in accordance with the eyesight of different people. According to weather forecasts (such as "visibility of 100 m"), we know the denseness that fog will be on particular days, but we don't have an index for evaluating whether fog has been eliminated. Accordingly, we estimated the extent that the image has been improved by applying the defogging technology from the degree of expansion of the signal. As I just mentioned, to eliminate fog, the remaining signal is "expanded," and the extent to which visibility in fog is improved is determined from the extent that the signal is expanded.

In addition to the result obtained from the experiment, the extent of image enhancement to apply was determined by human eye. Aiming at surveillance, it is good that the subject of shooting becomes visible by defogging, but if too much defogging is applied, excessive noise becomes a problem. Accordingly, we determined the extent that the fog is eliminated while viewing the images with the camera users and personnel from operations and marketing divisions.

During the experiment, how did you generate fog indoors?

USUIWe used a humidifying device to reproduce foggy conditions. In fact, we initially had no idea how to recreate fog indoors, so we tried various ways like smoke machines for stage use. With fog suddenly everywhere, we might have caused trouble if it were mistaken for smoke. So, by way of experiment, we decided to see how much smoke could be emitted outdoors. In the end, though, we found that the humidifier was the best way.

HIROOKAHowever, an environment recreated indoors must surely be different from an actual foggy environment. In the end, we sought out actual environments in which fog is generated. This search led us to China, where many places with high latitudes and cold climates commonly experience fog. So we ended up performing our experiments in some of those places.

Toiling towards product commercialization

What were the key difficulties involved in the technical development of the surveillance camera?

Photo: USUI Tsutomu

USUIThe first difficulty was the difference between the laboratory and the outdoors. In the case of a laboratory, the light source is provided by the lighting fitted in the ceiling, and so on; on the other hand, outdoors, light is provided by the sun. The position of the sun, naturally, changes over the course of a day, so the outdoor brightness changes too. Moreover, outdoor scenes change all the time; that is, tree branches sway, cars pass by, and so on. Since it is not easy to sort out all the difficulties associated with recreating all outdoor scenes in the laboratory, in the end, as I just explained, we felt it was important to evaluate foggy images by going to actual sites and viewing scenes alongside customers and staff from our operations and marketing divisions. Indeed, by evaluating such scenes alongside our customers, we were able to find various problems that we would not have noticed by experimentation in the laboratory.

Photo: SAI Hirotomo

SAIFrom a technologic viewpoint, we faced a hurdle, namely, it is theoretically possible, but practically impossible. The camera we were studying had a predetermined size, so the scale of the circuits it could house was limited. As the circuits get bigger, their chips get bigger, and costs rise in accordance. Since the technology will be commercialized, circuit size must be consistent with costs as a matter of course. How to develop a circuit with appropriate scale was tough. In other words, we faced a conflict between cutting too many functions (and loosing users) and adding on functions (thereby increasing cost).

A lot of hard work was involved right up to product commercialization, wasn't there?

HIROOKAOne challenge concerning commercialization was so-called "robustness".

The environment in which the camera is used varies from customer to customer. For example, the resolution of the monitor and the frame rate at which images are output vary. Images becoming unstable when the environment shooting them changes must be prevented. Accordingly, it is necessary to check that the images are properly captured regardless of the conditions in which the customer is using the camera. However, when we try to exhaustively check these conditions, the task will become enormous and unfeasible. As a matter of fact, we considered the minimum number of combinations that should be checked in order to confirm no problems in regards to all conditions. We toiled hard investigating this checking method.

Towards a safer and more secure society

What do you think will be a role of surveillance cameras in the future?

Photo: SAI Hirotomo

SAIWe are aiming at a creating a camera with better vision than the human eye.

From the outset, surveillance cameras have existed to aid the human eye. At present, thanks to contrast enhancement, we have reached the stage that the camera can see just like the human eye does. What's more, aiming to create a safer and more secure society, in the years to come, we want to be able to capture images ranging from dark ones in which the human eye cannot see to bright ones, and also handle every other kind of weather (in addition to fog). And from the viewpoint of systems for assuring the safety and security of society, we want to apply our technology to products—such as in-vehicle cameras—other than surveillance cameras.

Figure 4: Future development of technologies

Photo: HIROOKA Shinichiro

HIROOKAThe technology that we described first, namely, contrast enhancement, remedies total underexposure (blackout) and total overexposure (whiteout). However, it is questionable whether the enhanced portions are actually what the customer wants to see. No matter how much backlighting in a portion of an image is remedied, if the figure targeted in that image does not appear, the technology is useless. On the contrary, it is possible that the portions surrounding the backlight-remedied portion will become relatively difficult to see. We must therefore "intelligently" judge what portions the customer wants to see and show only those portions to the customer. I think such a technology is probably one that provides better vision than the human eye. For example, if it is required to monitor faces, the technology must ensure that faces are detected and never lost. Moreover, if cars speeding away must be watched out for, it must recognize cars without fail.

One of our main goals is to create a surveillance camera that is intelligent and able to show images of places that the human eye cannot see.

Your passion for surveillance cameras has come across. All in all, you like cameras, don't you?

Photo: USUI Tsutomu

USUIYes, I do. In addition to developing surveillance cameras, I am currently in charge of developing the technology for in-vehicle cameras. Being interested in cameras as well as cars, I am lucky enough to be involved in developing products of interest to me. In a similar manner, in-vehicle cameras and surveillance cameras capture images in a variety of scenes in sunlight and in dark places like tunnels. Moreover, considered from the viewpoint that both people and cars must be recognized accurately, the basic technology for in-vehicle cameras is the same as that for surveillance cameras. I want to develop the technology for in-vehicle cameras in the similar manner as that for surveillance cameras.

HIROOKAAs for me, since my university days, I have been interested in image recognition and image processing. My research theme at now is how can we, as researchers, create products that will contribute to society. In other words, I want to continue developing technologies that will contribute to creating a safe and secure society.

(Publication: February 26, 2013)

Notification

  • Publication: February 26, 2013
  • Professional affiliation and official position are at the time of publication.
  • Page top

Related contents

  • Page top