All major mobile operators are actively promoting their 5G advancements – If you believe the commercials, 5G is being rolled out and turned up, enabling higher bandwidth applications and machine-to-machine int...
Infotainment design is more critical than ever as autonomous vehicle functions are integrated into displays to keep drivers and passengers constantly informed of their surroundings. Dashboards have come a long way ...
Each of myDevices "IoT-in-a-box" kits contains the hardware needed to address a specific application. The world has been embroiled in aggressive technology progression since the creation of the steam en...
I’m not a cynic, but I know I come across that way sometimes. It’s more that I’m skeptical, because I continually hear, “it’s the best thing ever” or some form of that. And even w...
The DSP revolutionized the field of signal processing back in the 1980s by reducing noise, improving accuracy, and easing programming for engineers working with analog signals. Over the next few decades, DSPs advanc...
Too many organizations fail to implement effective data quality and risk management policies. When data comes in, they typically validate and cleanse it first before distributing it more widely. The emphasis i...
The world is full of bright and creative ideas for innovative and useful products. Brilliant engineers and marketers are eager to bring these products customers who will purchase them and appreciate the way they enh...
Measuring current to determine battery status or power consumed by a load such as a motor, lamp, or subsystem is a common circuit requirement. While there are various ways to do this, such as using a Hall-effect sen...
Wireless charging is becoming all the rage lately, especially when it comes to smartphones. But imagine if you were able to wirelessly charge your electric vehicle? Can it be possible? In some cars, it’s alrea...
Here’s something I’ve never thought about before: an IoT rose. No, I’m not talking about a bouquet of flowers that are able to transmit and receive data. I’m saying it’s possible to gro...
The USB Type-C specification was published in 2014, promising a significant increase in bandwidth and power delivery capability. Five years later and with USB Type-C solutions gaining critical mass, Embedded Computi...
One thing that I’ve learned about ams is that the company is all about sensors. If you need to sense something, whether it’s consumer, industrial health care, automotive, or something else, there’s...
In a relatively short amount of time, designing a home-automation platform has gone from an unenviable task to something that’s difficult for the seasoned developers. As new platforms roll out, the bigger ques...
As far as I can tell, Ambiq Micro is already the leader in low-power MCUs. I offer proof of that—my step-counter watch built with an Ambiq processor needs a new battery around once a year. I find that pretty a...
Achronix Semiconductor, a developer of FPGA-based hardware accelerators, recently announced the availability of its Speedcore Gen4 embedded FPGA (eFPGA) IP. The modular IP is aimed at compute, networking and st...
The smart home market has been seemingly poised for takeoff for decades. As we approach 2019, Avi Barel, Director of Business Development at the ULE Alliance, gives his perspective on what's working, what's ...
Since 2006, the OPC Unified Architecture (OPC UA) has provided a multivendor, multiplatform information modeling and communications architecture for industrial networks. The machine-to-machine (M2M) abstract layer i...
The Qt Company’s annual World Summit hit Berlin last week, a city that’s become increasingly popular for start-up tech companies. The messages delivered were clear: no longer can User Experience (UX) be ...
GE’s Intelligent Platforms group recently unveiled two new control solutions, the CPL410, an outcome-optimizing open controller, and a remote-monitoring service platform, the Equipment Insight 2018. As part of...
AUTOSAR was developed in 2003 to address the explosion of automotive software, defining an open industry-standard for electronic vehicle architectures. Over the years AUTOSAR has evolved to meet emerging automotive ...
I recently attended a press event where a dozen or so companies pitched their latest technologies. The presenting organizations ranged from semiconductor IP providers to component distributors, from connector compan...
First there was the basic step counter. Then gradually over time, these wearable devices got smarter and smarter. And that trend will continue thanks in part to the release of Maxim’s Health Sensor Platform 2....
“The Mostly Complete Chart of Neural Networks, Explained” was published in August 2017, and identified 27 different types of neural networks. The number of neural network types has increased since then a...
Although the IIoT is poised to reshape factory automation, transportation, energy, and other industrial markets through the application of intelligence and connectivity, the embedded systems that power these segment...
Buffer overflows have been the most commonly exploited vulnerability in network-borne attacks over the last 30 years. This isn’t surprising given how buffers are created. Here is an example in C: Step 1. Pro...
Current trends in the car industry mean that developing software projects has become an increasingly complex task. Software teams and project managers require specialized error search and process optimization framew...
With the expectation that within just a few years the number of deployed IoT nodes will be measured in many tens of billions, ensuring that sufficient security is installed represents a challenge of genuinely huge p...
The open LoRaWAN standard for IoT low-power wide area networks (LPWANs) enables smart sensors, devices, and other objects to connect wirelessly via gateways that relay messages to a central network server. It is kno...
Chances and challenges for machine learning in highly automated driving, part 3: Algorithm selection criteria for functional safety
In part one of this three-part series, the authors investigate the drivers behind and potential applications of machine learning technology in highly automated driving scenarios. Part two defines the theor...
The art of designing efficient and compact DC-to-DC converters is practiced by a select group of engineers with a deep understanding of the physics and supporting mathematics involved in conversion design, combined ...
These techniques are based on measuring phase differences between received radio-frequency (RF) signals and numerically computing AoA or AoD based on these phase differences. By using the resulting angle readings, d...
A variety of smart home automation solutions are being developed nowadays, but most of them lack the potential for seamless integration into an already functioning home environment aa well as security. To bridge the...
Chances and challenges for machine learning in highly automated driving, part 2: Theoretical background
In part one of this three-part series, the authors investigate the drivers behind and potential applications of machine learning technology in highly automated driving scenarios. Part two defines the theoretical bac...
From logistics regression to self-driving cars: Chances and challenges for machine learning in highly automated driving
In part one of this three-part series, the authors investigate the drivers behind and potential applications of machine learning technology in highly automated driving scenarios. Part two defines the theoretical background of machine learning technology, as well as the types of neural networks available to automotive developers. Part three evaluates these options in the context of functional safety requirements.
Machine learning has been one of the hottest topics in research and industry over the last couple of years. Renewed attention has resulted from the latest advancements in computational performance and algorithms compared to the advent of machine learning decades ago.
Recent impressive results in artificial intelligence have been facilitated by machine learning, and particularly deep learning solutions. Applications include natural language processing (NLP), personal assistance, the victory of AlphaGo over a human being, and the achievement of human-level behavior in learning to play Atari games.
Considering that machine learning and deep learning enable such impressive results when tackling extremely complex problems, it is obvious that researchers and engineers have considered also applying them to highly automated driving (HAD) scenarios in self-driving cars. The first promising results have been achieved in this area with NVIDIA’s Davenet, Comma.Ai, Google Car, and Tesla. Machine learning and deep learning approaches have resulted in initial prototypes, but the industrialization of such functionalities poses additional challenges with regard to, for example, essential functional safety considerations.
This article aims to contribute to ongoing discussions about the role of machine learning in the automotive industry and to highlight the importance of this topic in the context of self-driving cars. In particular, it aims to increase understanding of the capabilities and limitations of machine learning technologies.
First, we discuss the design space and architectural alternatives for machine learning-based highly automated driving in the context of the EB robinos reference architecture. Two selected use cases currently in research and development at Elektrobit are then presented in detail.
The second installment provides theoretical background to machine learning and deep neural networks (DNN) that provide the basis for deriving criteria used to select a machine learning technology according to a given task. Finally, the third installment discusses verification and validation challenges that affect functional safety considerations.
Machine learning and highly automated driving
It is a complex and non-trivial task to develop the highly automated driving functionalities that lead to self-driving cars. Engineers typically tackle such challenges using the principle of divide and conquer. This is for a good reason: A decomposed system with clearly defined interfaces can be tested and verified much more thoroughly than a single black box.
Our approach to highly automated driving is EB robinos, depicted in Figure 1. EB robinos is a functional software architecture with open interfaces and software modules that permits developers to manage the complexity of autonomous driving. The EB robinos reference architecture integrates components following the "Sense, Plan, Act" decomposition paradigm. Moreover, it makes use of machine learning technology within its software modules in order to cope with highly unstructured real-world driving environments. The subsections below contain selected examples of the technologies that are integrated within EB robinos.
Figure 1. Open EB robinos reference architecture.
In contrast, end-to-end deep learning approaches also exist, which span everything from sense to act (Bojarski et al. 2016). However, with respect to the handling and training of corner cases and rare events, and with regards to the exponential amount of training data necessary, a decomposition approach (i.e., semantic abstraction) is considered as more reasonable (Shalev- Shwartz et al. 2016).
Nevertheless, a decision about which parts are better tackled in isolation to others or in combination with others is required even if the decomposition approach is followed. It is also necessary to determine whether a machine learning approach is expected to outperform a traditionally engineered algorithm for the task accomplished by a particular block. Not least, this decision may be influenced by functional safety considerations. Functional safety is a crucial element of autonomous driving, as described later in this series. Traditional software components are written on the basis of concrete requirements and are tested accordingly.
The main issues in the testing and validation of machine learning systems are their black box nature and the stochastic behavior of the learning methods. It is basically impossible to predict how the system learns its structure.
The criteria and theoretical background given above can provide guidance for informed decisions. Elektrobit is currently researching and developing use cases in which machine learning approaches are considered to be promising. Two such use cases are presented next. The first deals with the generation of artificial training samples for machine learning algorithms and their deployment for traffic sign recognition. The second use case describes our approach to self-learning cars. Both examples make use of current cutting-edge deep learning technology.
Use case 1: Artificial sample generation and traffic sign recognition
This project proposes a speed limit and end of restriction traffic sign (TS) recognition system in the context of enhancing OpenStreetMap (OSM) data used in entry navigation systems. The aim is to run the algorithm on a standard smartphone that can be mounted on the windshield of a car. The system detects traffic signs along with their GPS position and uploads the collected data to backend servers via the mobile data connection of the phone. The approach is divided mainly into two stages: detection and recognition. Detection is achieved through a boosting classifier. Recognition is performed through a probabilistic Bayesian inference framework that fuses information delivered by a collection of visual probabilistic filters. The next installment of this article contains a description of the theoretical background behind the used algorithms. Figure 2 depicts the block diagram of the traffic signs recognition (TSR) algorithm.
Figure 2: Block diagram of the smartphone-based TSR system
The color image obtained is passed to the detector in 24-bit RGB format. The detection process is carried out by evaluating the response of a cascade classifier calculated through a detection window.
This detection window is shifted across the image at different scales. The probable traffic sign regions of interest (RoI) are collected as a set of object hypotheses. The classification cascade is trained with extended local binary patterns (eLPB) from the point of view of feature extraction. Each element in the hypotheses vector is classified into a traffic sign by a support vector machine (SVM) learning algorithm.
Traffic sign recognition methods rely on manually labelled traffic signs, which are used to train both the detection and the recognition classifiers. The labelling process is tedious and prone to error due to the variety of traffic sign templates used in different countries. Figure 3 shows the differences in some speed limit signs.
Figure 3. Traffic sign templates for different countries [source: www.wikipedia.org]. (a) Vienna convention. (b) United Kingdom. (c) Alternative Vienna convention. (d) Ireland. (e) Japan. (f) Samoa. (g) United Arab Emirates and Saudi Arabia. (h) Canada. (i) United States. (j) United States (Oregon variant).
Specific training data for each country is required for the traffic sign recognition method to perform well. It is time-consuming to create enough manually labelled traffic signs because position, illumination, and weather conditions have to be taken into account.
Elektrobit therefore has created an algorithm that generates training data automatically from a single artificial template image to overcome the challenge of manually annotating large numbers of training samples. Figure 4 shows the structure of the algorithm.
Figure 4. Block diagram of the artificial samples generation algorithm for machine learning-based recognition systems.
This approach provides a method for generating artificial data that is used in the training stages of machine learning algorithms. The method uses the reduced dataset of real and generic traffic sign image templates for each country to output a collection of images.
The features of these images are artificially defined by a sequence of image template deformation algorithms. The artificial images thus obtained are evaluated against a reduced set of real-world images using kernel principal components analysis (KPCA). The artificial data set is suitable for the training of machine learning systems, in this particular case for traffic sign recognition, when the characteristics of the generated images correspond to those of the real images.
Elektrobit replaced the Boosting SVM classifiers with a deep region-based detection and recognition convolutional neural network to improve the precision of the original traffic sign recognition system. The network is deployed using Caffe (Jia et al. 2014), which is a deep neural network library developed by Berkley and supported by NVIDIA. Caffe is a pure C++/CUDA library with Python and Matlab interfaces. In addition to its core deep learning functionalities, Caffe also provides reference deep learning models that can be used directly in machine learning applications. Figure 5 shows the Caffe net structure used for traffic sign detection and recognition. The different, colored blocks represent convolution (red), pooling (yellow), activation (green), and fully connected network layers (purple).
Figure 5. Deep region-based detection and recognition convolutional neural network in Caffe.
Use case 2: Learning how to drive
The revolution in deep learning has recently increased attention on another paradigm, which is referred to as reinforcement learning (RL). In RL, an agent by itself learns how to perform certain tasks by means of a reward system. The methodology is in the category of semi-supervised learning because the design of the reward system requires domain-specific knowledge.
That is even though there is no required labeling for the input data, in contrast with supervised learning. This recent interest in RL is due mainly to the seminal work of the Deep Mind team. This team managed to combine RL with a deep neural network capable of learning the action value function (Mnih et al. 2016). Their system was able to learn to play several Atari games at human-level capacity.
We constructed the deep reinforcement learning system, shown in Figure 6, in order to experiment safely with autonomous driving learning. This system uses the TORCS open-source race simulator (Wymann et al. 2014). TORCS is widely used in the scientific community as a highly portable multi-platform car-racing simulator. It runs on Linux (all architectures, 32- and 64-bit, little and big endian), FreeBSD, OpenSolaris, MacOSX, and Windows (32- and 64-bit). It features many different cars, tracks, and opponents to race against. We can collect images for object detection as well as critical driving indicators from the game engine. These indicators include the speed of the car, the relative position of the ego-car to the center line of the road, and the distances to the cars in front.
Figure 6. Deep reinforcement learning architecture for learning how to drive in a simulator.
The goal of the algorithm is to self-learn driving commands by interacting with the virtual environment. A deep reinforcement learning paradigm was used for this purpose, in which a deep convolutional neural network (DNN) is trained by reinforcing actions a that provide a positive reward signal
r(s^',a). The state
sis represented by the current game image as seen in the simulator window. There are four possible actions: accelerate, decelerate, turn left, and turn right.
The DNN computes a so-called
Q-function, which predicts the optimal action a to be executed for a specific state
s. In other words, the DNN calculates a
Q-value for each state-action pair. The action with the highest
Q-value will be executed, which moves the simulator environment to the next state,
s'. In this state, the executed action is evaluated by means of the reward signal
For example, if the car was able to accelerate without a collision, the related action that made this possible will be reinforced in the DNN; otherwise, it will be discouraged. The reinforcement is performed in the framework by retraining the DNN with the state-reward signals. Figure 7 shows the Caffe implementation for the deep reinforcement learning algorithm. The network layers have the same color-coding as in Figure 6.
Figure 7. A Caffe-based deep convolutional neural network structure used for deep reinforcement learning.
Part two defines the theoretical background of machine learning technology, as well as the types of neural networks available to automotive developers.
Associate Professor Sorin Mihai Grigorescu received his Dipl.-Eng. degree in Control Engineering and Computer Science from the Transylvania University of Brasov, Romania in 2006, and his Ph.D. in Robotics from the University of Bremen, Germany, in 2010. Between 2006 and 2010 he was a member of the Institute of Automation at the University of Bremen. Since June 2010, he has been affiliated with the Department of Automation at Transylvania University of Brasov, where he has been leading the Robust Vision and Control Laboratory. Since June 2013, he has also been affiliated with Elektrobit Automotive Romania, where he is the team manager of the Navigation Department. Sorin M. Grigorescu was an exchange researcher at several institutes, such as the Intelligent Autonomous Systems Group (Technical University Munich), the Korea Advanced Institute of Science and Technology (KAIST), and the Robotic Intelligence Lab in University Jaume I (Spain). Sorin was the winner of the EB innovation award 2013 for his work on machine learning algorithms used to build next-generation adaptable software for cars.
Markus Glaab is an expert in automotive software architectures at EB Automotive Consulting. In 2010, he received his M.Sc. degree in Computer Science from the University of Applied Sciences Darmstadt, Germany, while gaining professional experience as a software developer in the area of Car2X communication. In 2010, he also started research at both the In-Car-Multimedia Labs at his home university, where he worked on future service delivery platforms for vehicle-to-backend architectures, and the Centre for Security, Communications, and Network Research at Plymouth University, UK. Markus has been with EB since 2016 and works on future automotive E/E architectures and the integration of related technologies such as machine learning.
André Roßbach is a senior expert for functional safety at EB Automotive Consulting. In 2003, he received his degree in Business Information Technology from the University of Applied Sciences Hof. He then gained experience as a software developer in the biometric and distributed database development area. André has been with EB since 2004 and has developed software for medical systems, navigation software, and driver assistance systems. Currently, his focus is on functional safety, agile development, and machine learning.
This article is the first in a series devoted to the cloud technologies market and the role of fog computing for its participants. Today, the opportunities presented by cloud services and data storage are transform...
Why closed-loop sensing provides the best white color-tuning performance in LED lights at the lowest cost
The lighting industry has recognized the value of color tuning to support added-value applications such as human-centric lighting. There is a stark difference between the three models of color tuning control in use ...
Today, several publications and other media outlets writing about how to apply blockchain in fog computing mostly focus on the features directly brought by blockchain, such as secure identification, immutability, da...
This blog is part two in a three part series. Read part one here. The AGL Software Defined Car Architecture white paper defines how the AGL target platform for software defined vehicles can be implemented by using ...
This is part four in a series of excerpts from a book written by Roderick Chapman and Yannick Moy that cover AdaCore Technologies for Cyber Security. This part presents a number of security-related scenari...
The hockey stick in adoption of artificial intelligence (AI) applications is well under way and custom architectures including CPUs, GPUs, FPGAs, and neural network processors are claiming their place in the cloud. ...