Snow Hazard Index Using Conditional GAN and Semantic Segmentation

Recently, there was a record heavy snowfall due to climate change. In Japan, thousands of vehicles were stuck on the highway for three days. Because of the freezing of the road surface, there was a multi-vehicle accident. Road managers are required to provide indicators to alert drivers regarding snow cover at hazardous locations. During the night, the temperature drops and the road surface tends to freeze. Road managers are required to make decisions on road closures and snow removal work based on the road surface conditions at night.

This session proposes a custom loop deep learning application with live image post-processing to automatically calculate a snow hazard indicator. First, we translate the road surface hidden under snow using a generative adversarial network (GAN), pix2pix. Second, we detect snow-covered and road surface classes by semantic segmentation using DeepLabv3+ with MobileNet as a backbone. Third, we prepare one-to-one paired images and develop a snowy night-to-day translator from the night snow image to the day fake output using the Conditional GAN. Based on these trained networks by MATLAB^® toolboxes, we can automatically compute the road to snow rate hazard index, indicating the amount of snow covered on the road surface, even at night.

We demonstrate the applied results to thousands of live snow images of the cold region in Japan. This application has the advantage of using only one live image as the input, without any mining of the before and after paired images dataset. Furthermore, the indicator could be delivered to the road managers and users using our pipeline automatically computed snow hazard ratio index, whose value was zero to 100 for multi-points comparison. As a result, the fake day label output of snowy night images has been well approximated to the real label of snowy day per pixel, with the critical ROI as the snow category for monitoring the winter road safety.