The open LoRaWAN standard for IoT low-power wide area networks (LPWANs) enables smart sensors, devices, and other objects to connect wirelessly via gateways that relay messages to a central network server. It is kno...
Chances and challenges for machine learning in highly automated driving, part 3: Algorithm selection criteria for functional safety
In part one of this three-part series, the authors investigate the drivers behind and potential applications of machine learning technology in highly automated driving scenarios. Part two defines the theor...
The art of designing efficient and compact DC-to-DC converters is practiced by a select group of engineers with a deep understanding of the physics and supporting mathematics involved in conversion design, combined ...
These techniques are based on measuring phase differences between received radio-frequency (RF) signals and numerically computing AoA or AoD based on these phase differences. By using the resulting angle readings, d...
A variety of smart home automation solutions are being developed nowadays, but most of them lack the potential for seamless integration into an already functioning home environment aa well as security. To bridge the...
Chances and challenges for machine learning in highly automated driving, part 2: Theoretical background
In part one of this three-part series, the authors investigate the drivers behind and potential applications of machine learning technology in highly automated driving scenarios. Part two defines the theoretical bac...
From logistics regression to self-driving cars: Chances and challenges for machine learning in highly automated driving
In part one of this three-part series, the authors investigate the drivers behind and potential applications of machine learning technology in highly automated driving scenarios. Part two defines the theoretical background of machine learning technology, as well as the types of neural networks available to automotive developers. Part three evaluates these options in the context of functional safety requirements.
Machine learning has been one of the hottest topics in research and industry over the last couple of years. Renewed attention has resulted from the latest advancements in computational performance and algorithms compared to the advent of machine learning decades ago.
Recent impressive results in artificial intelligence have been facilitated by machine learning, and particularly deep learning solutions. Applications include natural language processing (NLP), personal assistance, the victory of AlphaGo over a human being, and the achievement of human-level behavior in learning to play Atari games.
Considering that machine learning and deep learning enable such impressive results when tackling extremely complex problems, it is obvious that researchers and engineers have considered also applying them to highly automated driving (HAD) scenarios in self-driving cars. The first promising results have been achieved in this area with NVIDIA’s Davenet, Comma.Ai, Google Car, and Tesla. Machine learning and deep learning approaches have resulted in initial prototypes, but the industrialization of such functionalities poses additional challenges with regard to, for example, essential functional safety considerations.
This article aims to contribute to ongoing discussions about the role of machine learning in the automotive industry and to highlight the importance of this topic in the context of self-driving cars. In particular, it aims to increase understanding of the capabilities and limitations of machine learning technologies.
First, we discuss the design space and architectural alternatives for machine learning-based highly automated driving in the context of the EB robinos reference architecture. Two selected use cases currently in research and development at Elektrobit are then presented in detail.
The second installment provides theoretical background to machine learning and deep neural networks (DNN) that provide the basis for deriving criteria used to select a machine learning technology according to a given task. Finally, the third installment discusses verification and validation challenges that affect functional safety considerations.
Machine learning and highly automated driving
It is a complex and non-trivial task to develop the highly automated driving functionalities that lead to self-driving cars. Engineers typically tackle such challenges using the principle of divide and conquer. This is for a good reason: A decomposed system with clearly defined interfaces can be tested and verified much more thoroughly than a single black box.
Our approach to highly automated driving is EB robinos, depicted in Figure 1. EB robinos is a functional software architecture with open interfaces and software modules that permits developers to manage the complexity of autonomous driving. The EB robinos reference architecture integrates components following the "Sense, Plan, Act" decomposition paradigm. Moreover, it makes use of machine learning technology within its software modules in order to cope with highly unstructured real-world driving environments. The subsections below contain selected examples of the technologies that are integrated within EB robinos.
Figure 1. Open EB robinos reference architecture.
In contrast, end-to-end deep learning approaches also exist, which span everything from sense to act (Bojarski et al. 2016). However, with respect to the handling and training of corner cases and rare events, and with regards to the exponential amount of training data necessary, a decomposition approach (i.e., semantic abstraction) is considered as more reasonable (Shalev- Shwartz et al. 2016).
Nevertheless, a decision about which parts are better tackled in isolation to others or in combination with others is required even if the decomposition approach is followed. It is also necessary to determine whether a machine learning approach is expected to outperform a traditionally engineered algorithm for the task accomplished by a particular block. Not least, this decision may be influenced by functional safety considerations. Functional safety is a crucial element of autonomous driving, as described later in this series. Traditional software components are written on the basis of concrete requirements and are tested accordingly.
The main issues in the testing and validation of machine learning systems are their black box nature and the stochastic behavior of the learning methods. It is basically impossible to predict how the system learns its structure.
The criteria and theoretical background given above can provide guidance for informed decisions. Elektrobit is currently researching and developing use cases in which machine learning approaches are considered to be promising. Two such use cases are presented next. The first deals with the generation of artificial training samples for machine learning algorithms and their deployment for traffic sign recognition. The second use case describes our approach to self-learning cars. Both examples make use of current cutting-edge deep learning technology.
Use case 1: Artificial sample generation and traffic sign recognition
This project proposes a speed limit and end of restriction traffic sign (TS) recognition system in the context of enhancing OpenStreetMap (OSM) data used in entry navigation systems. The aim is to run the algorithm on a standard smartphone that can be mounted on the windshield of a car. The system detects traffic signs along with their GPS position and uploads the collected data to backend servers via the mobile data connection of the phone. The approach is divided mainly into two stages: detection and recognition. Detection is achieved through a boosting classifier. Recognition is performed through a probabilistic Bayesian inference framework that fuses information delivered by a collection of visual probabilistic filters. The next installment of this article contains a description of the theoretical background behind the used algorithms. Figure 2 depicts the block diagram of the traffic signs recognition (TSR) algorithm.
Figure 2: Block diagram of the smartphone-based TSR system
The color image obtained is passed to the detector in 24-bit RGB format. The detection process is carried out by evaluating the response of a cascade classifier calculated through a detection window.
This detection window is shifted across the image at different scales. The probable traffic sign regions of interest (RoI) are collected as a set of object hypotheses. The classification cascade is trained with extended local binary patterns (eLPB) from the point of view of feature extraction. Each element in the hypotheses vector is classified into a traffic sign by a support vector machine (SVM) learning algorithm.
Traffic sign recognition methods rely on manually labelled traffic signs, which are used to train both the detection and the recognition classifiers. The labelling process is tedious and prone to error due to the variety of traffic sign templates used in different countries. Figure 3 shows the differences in some speed limit signs.
Figure 3. Traffic sign templates for different countries [source: www.wikipedia.org]. (a) Vienna convention. (b) United Kingdom. (c) Alternative Vienna convention. (d) Ireland. (e) Japan. (f) Samoa. (g) United Arab Emirates and Saudi Arabia. (h) Canada. (i) United States. (j) United States (Oregon variant).
Specific training data for each country is required for the traffic sign recognition method to perform well. It is time-consuming to create enough manually labelled traffic signs because position, illumination, and weather conditions have to be taken into account.
Elektrobit therefore has created an algorithm that generates training data automatically from a single artificial template image to overcome the challenge of manually annotating large numbers of training samples. Figure 4 shows the structure of the algorithm.
Figure 4. Block diagram of the artificial samples generation algorithm for machine learning-based recognition systems.
This approach provides a method for generating artificial data that is used in the training stages of machine learning algorithms. The method uses the reduced dataset of real and generic traffic sign image templates for each country to output a collection of images.
The features of these images are artificially defined by a sequence of image template deformation algorithms. The artificial images thus obtained are evaluated against a reduced set of real-world images using kernel principal components analysis (KPCA). The artificial data set is suitable for the training of machine learning systems, in this particular case for traffic sign recognition, when the characteristics of the generated images correspond to those of the real images.
Elektrobit replaced the Boosting SVM classifiers with a deep region-based detection and recognition convolutional neural network to improve the precision of the original traffic sign recognition system. The network is deployed using Caffe (Jia et al. 2014), which is a deep neural network library developed by Berkley and supported by NVIDIA. Caffe is a pure C++/CUDA library with Python and Matlab interfaces. In addition to its core deep learning functionalities, Caffe also provides reference deep learning models that can be used directly in machine learning applications. Figure 5 shows the Caffe net structure used for traffic sign detection and recognition. The different, colored blocks represent convolution (red), pooling (yellow), activation (green), and fully connected network layers (purple).
Figure 5. Deep region-based detection and recognition convolutional neural network in Caffe.
Use case 2: Learning how to drive
The revolution in deep learning has recently increased attention on another paradigm, which is referred to as reinforcement learning (RL). In RL, an agent by itself learns how to perform certain tasks by means of a reward system. The methodology is in the category of semi-supervised learning because the design of the reward system requires domain-specific knowledge.
That is even though there is no required labeling for the input data, in contrast with supervised learning. This recent interest in RL is due mainly to the seminal work of the Deep Mind team. This team managed to combine RL with a deep neural network capable of learning the action value function (Mnih et al. 2016). Their system was able to learn to play several Atari games at human-level capacity.
We constructed the deep reinforcement learning system, shown in Figure 6, in order to experiment safely with autonomous driving learning. This system uses the TORCS open-source race simulator (Wymann et al. 2014). TORCS is widely used in the scientific community as a highly portable multi-platform car-racing simulator. It runs on Linux (all architectures, 32- and 64-bit, little and big endian), FreeBSD, OpenSolaris, MacOSX, and Windows (32- and 64-bit). It features many different cars, tracks, and opponents to race against. We can collect images for object detection as well as critical driving indicators from the game engine. These indicators include the speed of the car, the relative position of the ego-car to the center line of the road, and the distances to the cars in front.
Figure 6. Deep reinforcement learning architecture for learning how to drive in a simulator.
The goal of the algorithm is to self-learn driving commands by interacting with the virtual environment. A deep reinforcement learning paradigm was used for this purpose, in which a deep convolutional neural network (DNN) is trained by reinforcing actions a that provide a positive reward signal
r(s^',a). The state
sis represented by the current game image as seen in the simulator window. There are four possible actions: accelerate, decelerate, turn left, and turn right.
The DNN computes a so-called
Q-function, which predicts the optimal action a to be executed for a specific state
s. In other words, the DNN calculates a
Q-value for each state-action pair. The action with the highest
Q-value will be executed, which moves the simulator environment to the next state,
s'. In this state, the executed action is evaluated by means of the reward signal
For example, if the car was able to accelerate without a collision, the related action that made this possible will be reinforced in the DNN; otherwise, it will be discouraged. The reinforcement is performed in the framework by retraining the DNN with the state-reward signals. Figure 7 shows the Caffe implementation for the deep reinforcement learning algorithm. The network layers have the same color-coding as in Figure 6.
Figure 7. A Caffe-based deep convolutional neural network structure used for deep reinforcement learning.
Part two defines the theoretical background of machine learning technology, as well as the types of neural networks available to automotive developers.
Associate Professor Sorin Mihai Grigorescu received his Dipl.-Eng. degree in Control Engineering and Computer Science from the Transylvania University of Brasov, Romania in 2006, and his Ph.D. in Robotics from the University of Bremen, Germany, in 2010. Between 2006 and 2010 he was a member of the Institute of Automation at the University of Bremen. Since June 2010, he has been affiliated with the Department of Automation at Transylvania University of Brasov, where he has been leading the Robust Vision and Control Laboratory. Since June 2013, he has also been affiliated with Elektrobit Automotive Romania, where he is the team manager of the Navigation Department. Sorin M. Grigorescu was an exchange researcher at several institutes, such as the Intelligent Autonomous Systems Group (Technical University Munich), the Korea Advanced Institute of Science and Technology (KAIST), and the Robotic Intelligence Lab in University Jaume I (Spain). Sorin was the winner of the EB innovation award 2013 for his work on machine learning algorithms used to build next-generation adaptable software for cars.
Markus Glaab is an expert in automotive software architectures at EB Automotive Consulting. In 2010, he received his M.Sc. degree in Computer Science from the University of Applied Sciences Darmstadt, Germany, while gaining professional experience as a software developer in the area of Car2X communication. In 2010, he also started research at both the In-Car-Multimedia Labs at his home university, where he worked on future service delivery platforms for vehicle-to-backend architectures, and the Centre for Security, Communications, and Network Research at Plymouth University, UK. Markus has been with EB since 2016 and works on future automotive E/E architectures and the integration of related technologies such as machine learning.
André Roßbach is a senior expert for functional safety at EB Automotive Consulting. In 2003, he received his degree in Business Information Technology from the University of Applied Sciences Hof. He then gained experience as a software developer in the biometric and distributed database development area. André has been with EB since 2004 and has developed software for medical systems, navigation software, and driver assistance systems. Currently, his focus is on functional safety, agile development, and machine learning.
This article is the first in a series devoted to the cloud technologies market and the role of fog computing for its participants. Today, the opportunities presented by cloud services and data storage are transform...
Why closed-loop sensing provides the best white color-tuning performance in LED lights at the lowest cost
The lighting industry has recognized the value of color tuning to support added-value applications such as human-centric lighting. There is a stark difference between the three models of color tuning control in use ...
Today, several publications and other media outlets writing about how to apply blockchain in fog computing mostly focus on the features directly brought by blockchain, such as secure identification, immutability, da...
This blog is part two in a three part series. Read part one here. The AGL Software Defined Car Architecture white paper defines how the AGL target platform for software defined vehicles can be implemented by using ...
This is part four in a series of excerpts from a book written by Roderick Chapman and Yannick Moy that cover AdaCore Technologies for Cyber Security. This part presents a number of security-related scenari...
The hockey stick in adoption of artificial intelligence (AI) applications is well under way and custom architectures including CPUs, GPUs, FPGAs, and neural network processors are claiming their place in the cloud. ...
The need for hand-held portable power sources has been driven by rapid growth in the use of smart phones, tablets, and other portable devices. According to the market research, the power bank market is expected to r...
In many real-time applications, the CPU can spend 95% (or more) of its time in less than 5% of the code. This is the case for motor control, engine control, wireless communications and many other time sensitive appl...
The high-speed ADC is a primary design consideration in all wideband EW receivers and largely determines system architecture and overall detect and observe capability. The ever-present need for higher bandwidth and...
Based on numerous studies, smart home and wearables are the most popular IoT applications today. Embedded MCUs are at the core of these IoT applications. However, to serve as an effective foundation in this fast and...
This piece is part one of a three part series and explores some of the trends and innovations that are emerging in the Industrial Internet of Things (IIoT) and the challenges they will help overcome. Standard solut...
From the outside, few things could be simpler than a USB stick or thumb drive, or an SD-format storage card. Each one contains a Flash memory array, a USB interface, and a storage controller that provides generic su...
Microelectromechanical systems (MEMS) are devices made in specialty microelectronic fabs that combine electronic functions and mechanical actions. MEMS, as a technology platform, have enabled the manufacture of uniq...
This is part two in a series of excerpts from a book written by Roderick Chapman and Yannick Moy that cover AdaCore Technologies for Cyber Security. It considers why building and operating secure computer ...
Today’s cybersecurity landscape is confusing and difficult to navigate. There are thousands of vendors selling different flavors of security solutions for embedded systems and networks. Solutions range from a...
The first wave of cloud-assisted applications for the IoT has demonstrated the value of the concept and the difference that advanced cooperative processing can make. These applications make it easier to detect patte...
Single-board computers (SBCs) and computer-on-module (CoM) systems embed all of the functions of a computer onto a single circuit board, and are commonly used in industrial applications in a rackmount format for fac...
Applications ranging from networking and cellular infrastructure to defense, commercial aviation, and industry 4.0 require significantly different features in FPGAs than is required for the data center market. While...
The Industrial Internet of Things (IIoT) is heralding a new wave of modernization, across many industries, with customers and internal stakeholders demanding advances in productivity, management, security,...
Some say that if you aren’t living on the edge, you’re taking too much room. Likewise, if you aren’t computing on the edge, you’re waiting too long for server response and consuming too much ...
This is part four of a four-part series. Read the other parts here: Part 1, Part 2, and Part 3. So far, we looked at what an MPU is and how it can help isolate tasks and processes from one another. We’v...
DIY electronics has emerged as one of the most effective means of educating young engineers. Jason Kridner, Co-Founder of BeagleBoard.org and a 2018 Top Embedded Innovator, explains how “Making” is makin...
As the Internet of Things (IoT) drives new levels of automation, distributed machines must be able to conduct transactions on their own. This requirement has given rise to blockchain, an inalterable distributed ledg...
Vehicles are increasingly software-defined networks on wheels. In response, Automotive Grade Linux (AGL) has exploded in popularity. Dan Cauchy, a 2018 Top Embedded Innovator and Executive Director of AGL offers ins...
This is part three of a four-part series. Read the other three parts here: Part 1, Part 2, and Part 4. So far, we looked at what an MPU is and how it can help isolate tasks and processes from one another. We&...
This is part three of a three part series. Read part two here. Recent advances in hardware and software have made it possible for compact, battery-powered products to include always-on voice command systems, which...
This is part two of a series. Read part one here. Network security While the edge receives most of the focus in IIoT, it is important to not neglect the cloud or the server side of a system. Test for common server...
We’ll be running a series of excerpts from a book written by Roderick Chapman and Yannick Moy that cover AdaCore Technologies for Cyber Security. Here’s the first, and it serves as an introduction to wha...
Introduced in 2004, the ARM Cortex-M architecture is currently the most popular 32-bit architecture on the market, adopted by most, if not all major MCU manufacturers. The Cortex-M was designed from the outset ...
IoT system attacks are making headlines and continue to showcase the security vulnerabilities of networks, edge nodes, and gateways. A recent Mirai botnet infected over 2.5 million IoT nodes by logging into devices ...
This is part two of a three part series. Read part one here. Recent advances in hardware and software have made it possible for compact, battery-powered products to include always-on voice command systems, which h...
This article is the first in a series. The other three parts can be found here: Part 2, Part 3, and Part 4. Memory-protection units (MPUs) have been available for years on processors such as the Cortex-M, and yet, ...
This is part one of a three part series. Catch part two next week. Recent advances in hardware and software have made it possible for compact, battery-powered products to include always-on voice command systems, wh...