There will be many cameras in our future cars. Rear view, surround view, mirror replacement, driver monitoring and front view ADAS applications all require one or more cameras. The wiring to hook up all these cameras to the electronic systems that are positioned elsewhere in the car can become quite complex. Replacing traditional wiring in our cars with the new 100BASE-T1 Ethernet standard has many advantages. This communication standard uses unshielded twisted pair cables to deliver data at a rate of 100Mbps. Due to the simpler wiring, smaller connectors, and a switched network, cost is reduced by 80 percent and weight by up to 30 percent compared to LVDS-based systems. We wrote an article on automotive Ethernet before, debunking the myths that it won’t work for ADAS applications. This time we’ll discuss the H.264-based video coding for automotive Ethernet in more detail.
Transmitting even a low-resolution 1.3Mpixel, 4:2:0, 8-bit, 30fps video stream consumes about 500Mbps of bandwidth. In order to ship this data over the 100Mbps 100BASE-T1 link, you need to compress the video. The most commonly used standard for video compression today is H.264. It has been around for over a decade. The first version was approved in 2003, and many SOCs support encoding or decoding H.264-coded bitstreams. Our phones, our PCs, our TVs, they all support H.264 for video capture or playback. No wonder the H.264 standard became the standard choice for automotive Ethernet also. So if your automotive camera supports H.264 and your head unit or ECU supports H.264 then you’re all set right? Unfortunately, things are not that simple. There are many variants of H.264, and different ways to implement an H.264 codec. Simply “supporting H.264” isn’t enough. Below you’ll find an overview of the many variants of H.264 and how especially the requirements for an H.264 codec for consumer and automotive applications are very different.
Profiles & levels
The H.264 standard defines many formats and algorithms to compress a video sequence. Profiles describe sets of these capabilities that a codec can support. A bitstream compressed by an H.264 encoder that conforms to one profile may not be able to be decoded by an H.264 decoder that conforms to a different profile. H.264 has at least 22 profiles, so it can be tricky to have the correct support. While profiles describe how the pixels are compressed, levels describe the performance in terms of picture resolution, frame rates and bit rates. Again, there are many levels, which are also dependent on the profile that’s selected.
1. Lower resolution
While consumer electronics such as TVs or mobile phones regularly support 4K resolution, and getting ready for 8K broadcast at the 2020 Olympics in Japan, automotive cameras are typically lower resolution. Especially in rear view and surround view systems, 1-2Mpixel imagers are typical.
2. Similar frame rates
Most consumer video is captured and displayed at 30fps today. There’s a trend toward higher frame rates. The iPhone, YouTube, as well as most 4K TVs today support 60fps. In automotive, it’s similar. Typical frame rates are 30fps today, but this is going up to 60fps. A 30fps camera mounted on a car that travels at typical highway speeds of 110km/hr moves about 1 meter per captured frame. Especially at higher speeds, higher frame rates are more important.
3. Higher bitrates
Consumer applications like video conferencing, snapchat, facetime and online video playback use fairly low bitrates that are below 10Mbps. Even set top boxes and high-bitrate Blu-ray discs stay well below 50Mbps. The 100BASE-T1-based automotive cameras use close to 100Mbps bandwidth, resulting in 2x-10x higher bitrates than what is used in typical consumer systems.
4. More video channels
In consumer applications there’s typically a single camera or single display in use at a time. In automotive there are usually many cameras and several displays involved. The autonomous driving or surround view systems process many of these high-bit-rate camera streams in parallel into a single view and deduce high-level data to control the car. The video coding system in such central units needs to be able to encode and decode many channels of H.264. The combined pixel rates and bitrates are in this case again an order of magnitude higher than consumer-based systems.
5. Lower delay
In most consumer applications low delay isn’t really important. Even in low-delay applications such as video conferencing a latency of around 150ms is acceptable. For automotive applications the delay needs to be limited as much as possible. Any delay causes downstream ADAS algorithms to respond more slowly. For rear and surround view systems, any delay causes the display in the car to lag behind what the driver sees going on around the vehicle. In short, while delay in consumer applications is no big deal, an H.264 implementation optimized for automotive should be as low delay as possible, down to less than 10 milliseconds.
6. Higher color depth
Virtually all consumer video applications support 8-bit color today. There’s a trend toward 10-bit, but it will take some time before the content and infrastructure is there to bring 10-bit video content to our mobile, PC and TV screens. The automotive industry has seen a clear need for 10-bit or even higher color depth to support the higher dynamic range for some time. Headlights of upcoming cars, light at the end of a tunnel or a rising sun all require the H.264 codec to support higher dynamic range.
7. Optimize picture quality for machine vision
The pictures from the cameras in cars are not only displayed on screens but also often forwarded to computer vision systems to analyze them. Since an H.264 codec is lossy, and some image data is lost, the compression artefacts may confuse machine vision algorithms downstream. There are many technical solutions to prevent this. All of them require the H.264 codec to be adapted toward such a scenario.
8. Lower memory usage
Automotive camera systems are severely memory constrained compared to mobile phones, tablets or TVs. Instead of running full-fledged operating systems and having access to gigabytes of memory, automotive cameras need to consume less power and be lower cost. Therefore they often don’t include DRAM, which adds cost and increases power. As a result an automotive H.264 codec must be optimized for minimal memory usage to function in these memory constrained devices.
9. Higher robustness
The ability to properly decode the H.264 stream even in case of transmission errors in automotive scenarios is key. For instance, losing a few frames of video when watching a movie can be annoying, but in automotive scenarios these situations can cause accidents.
10. Cover unexpected errors for automotive safety
The ISO 26262 functional safety standard requires conformance to reach specific Automotive Safety Integrity Levels (ASIL). Qualifying software components includes predicting software behavior in case of failure and overload situations. To qualify a software component, the standard requires regular testing as well as insertion of faults into the system to determine how it reacts to unexpected inputs. Erroneous behavior is then analyzed and countermeasures need to be implemented throughout the design process. This process requires an H.264 codec that can be adapted further during the design of the camera, ECU and system.
H.264 is a standard, but there’s no standard H.264 implementation. The requirements for H.264 in automotive ADAS systems are significantly different from the typical consumer video codec offering that are on the market today. Our H.264 solution has been optimized specifically for automotive applications and since our codec implementations are implemented in firmware running on our unified video/vision processor architecture, they can be upgraded during the design of the camera or ECU, and even upgraded in the field.