Testing and traceability play a vital role in embedded product development. Traceability rendered via Application Life-cycle Management (ALM) software enables the team to automatically generate test cases that link back to requirements, and report defects that link back to test cases. By knowing that defects are addressed and test cases run successfully, the team can have an immediate and accurate accounting of the state of product requirements.

Advances in static analysis and graphics technology have enabled new software visualization tools that can yield insights into the structure of complex programs, making the development process easier.

It has long been evident that pictures are often better than text in helping developers understand complex programs and review code, so the use of program visualization in software development is widespread. UML and other primarily graphical formal design notations are now widely accepted as the best standard mechanisms for communicating various aspects of software design. Some model-based design tools can generate code directly from graphical representations. At the informal end of the spectrum, developers often sketch out flowcharts or call graphs to inform themselves or others of important aspects of the software.

UML diagrams are all very well for designs, but suffer from two important drawbacks when used later in the development process to help developers understand existing code. First, as design abstractions they (correctly) omit some implementation details, but those details are often important if the goal is to understand the finished software. Second, design diagrams are very often stale with respect to the implementation, leading to an inaccurate or incomplete portrayal of the system as it actually exists.

Informal visualizations tend to be ephemeral and rarely make it into the official record of documentation for the program.

Very often, the only artifact that a developer has to work with is the code itself. Unfortunately, code visualization tools have historically been subject to problems such as confusing diagrams and difficulty in scaling to large programs. However, new tools are emerging that are beginning to solve these problems. The key advantage of these tools is their ability to generate useful visualizations directly from the code itself. As such they are guaranteed to be accurate and up to date.

Program structures

Programs are made up of a large and complex web of dependences between lots of different kinds of components. A visualization that attempted to show all of these simultaneously would be too unwieldy to be useful. Indeed, there is no single ideal visualization. Instead, the most useful visualization for a particular task is the one that corresponds to the mental model used by the engineer undertaking that task. Some of the more useful program structures are the following:

Type hierarchy

Developers usually find it very useful to see the various ways in which data types can relate to each other. The standard UML class diagram represents the class hierarchy in a form that is very easy to understand, with the association and containment relations at a higher level of abstraction than the code. While this is good from a design perspective, programmers often find it more helpful to see the concrete relations between types.

Include tree

C and C++ programs often can make heavy use of the preprocessor. If done well, this can make programs easy to understand, but very often it interposes a layer that gets in the way of understanding. Undisciplined use of the preprocessor can lead to dependence tangles that cause build problems and hurt reusability potential. Consequently, being able to see which files are included where can help engineers unravel complex dependences.

Call graph

The call graph, in which each node represents a subprogram and each edge indicates one or more calls to another subprogram, is often considered the most helpful program structure to visualize. Subprograms are convenient units for developers to reason about, and the calling relation captures data and control flow nicely. A call graph for even a small program can have hundreds of nodes and thousands of edges (see Sidebar 1), so it has long been recognized that it is essentially useless to visualize the entire call graph all at once. Instead, researchers have focused on ways to visualize the call graph in smaller, easily digestible parts.

New call graph techniques and tools

Because of the importance of call graphs in program understanding and the challenges involved in visualizing them, they have been the subject of much research. In particular, new techniques have been developed to help tame call graph complexity. This section describes some of the mechanisms that have been implemented in static analysis tools delivering advanced visualization capabilities.

Top-down views

A top-down view of a call graph helps answer user questions such as “What are the high-level components of this program, and what are their properties and relationships?”

To solve this problem in the context of program understanding, tool designers take inspiration from geographic mapping programs such as Google Maps. As the user zooms in, more detail starts to resolve: first cities, then towns, villages, and ultimately, individual buildings. The level of detail shown is coupled to the zoom level.

Programs are made up of components that are themselves made up of smaller components, and so on, forming a hierarchy; although the direct calling relationship is between low-level subprograms, it can be projected up to higher-level components that contain those subprograms. In the top-down view of a call graph, the highest level items are directories. These can contain some combination of subdirectories and files, and the files will then contain subprograms. Thus, an edge from one box to another simply indicates that a subprogram contained in the first box calls a subprogram contained in the second.

This approach turns out to be very effective at helping developers gain a deeper understanding of a program.

In the left window, the user has selected the edge from component find to component gnulib. The function calls summarized by this aggregate edge are shown in the pane to the right. The right window illustrates that more detail is shown when the user zooms in to see a single function. This zoom level further illustrates an important feature: It is important for the developer to be able to relate the view to the code itself. Consequently, selecting one of those functions causes the source code of that function to be shown.

Bottom-up views

Often a developer will want to take a bottom-up approach. This helps users answer questions such as “What does this procedure do, how does it fit into the structure of the program, and how is it invoked?”

For example, say a program has crashed in a particular function. To find the cause of the crash and to plan a fix, the developer is likely to begin by focusing on that single function, then explore its immediate neighborhood to see what other functions it calls and is called by. Previously done manually on a whiteboard, a tool can handle the drudgery of drawing and layout automatically.

Metrics layers

The utility of a visualization can be increased by adding layers to show the value of various metrics. An example is shown in Figure 1. This shows a particularly useful visualization – the treemap. In a treemap, the area of a node is proportional to a metric – usually a metric that encodes the size of the item. Subnodes are then tiled inside the top level node. Edges are usually not displayed. In this example, the color intensity of each item encodes the number of code vulnerability warnings issued by the static analysis tool.

From this view, it is easy to pick out the components of the program that are the most risky. Treemaps are very effective for showing deeply nested structure, and are also very amenable to the zooming paradigm discussed earlier, where more detail is shown at higher magnifications.

These visualizations are most useful when developers use them interactively to pan around and zoom in and out, or even add and remove nodes and edges. Interacting with such an interface can be extremely frustrating if it is not sufficiently responsive. Showing hundreds of nodes and thousands of edges can be a challenge.

Paul Anderson is VP of Engineering at GrammaTech. He received his B.Sc. from Kings College, University of London and his Ph.D. in Computer Science from City University London. Contact him at paul@grammatech.com.

GrammaTech, Inc. www.grammatech.com

Follow: Twitter Facebook Blog Google+ LinkedIn YouTube

When it comes to software development, the old adage is best spun in a slightly different way: better “early” than never. Accordingly, static analysis can help those developing in Java to stay one step ahead of potential coding problems.

Today’s software development teams are under immense pressure; the market demands high-quality, secure releases at a constantly increasing pace while security threats become more and more sophisticated. Considering the high cost of product failures and security breaches, it is more important than ever to address these risks throughout the software development process. Potential problems need to be spotted early to prevent release delays or, worse, post-release failures.

Fortunately, there are numerous tools to help developers manage these risks, helping to identify potential problems early in the development phase when issues are less disruptive and easier to fix. They are readily accessible to developers and easy to use within many development environments. This applies to developers programming in any language; however, we focus on Java in this discussion (see Sidebar 1).

Static analysis helps mitigate risk

When considering static analysis tools for Java or otherwise, it is important to understand what these tools are. The term “static analysis” refers to the approach of analyzing a program without executing it. As we’ll see in the next section, static analysis tools can be used to produce reports on anything from coding standard violations to specific errors or vulnerabilities. Simply put, static analysis tools analyze source code to find information useful for managing risk.

One benefit of static analysis is that it can be performed early in the development cycle, often before the application will even execute. It is commonly integrated into an automated build, so that there is virtually no overhead to running frequent analyses. By integrating static analysis into the inner development loop, users maximize the value they get from such tools.

When used in conjunction with a well-designed development process, static analysis tools provide crucial visibility into the state of the software. This enables development teams to understand the level of risk in their code and where the risk resides so they can take action to mitigate or remove it entirely (Table 1). Individual tools generally focus on specific problems faced by software development teams, and teams often use a combination of these tools to get a comprehensive view of their development effort.

Developers have traditionally used static analysis tools via a simple IDE integration or as stand-alone tools. While the tools add significant value to the development effort, the proliferation of tools has created efficiency problems as developers spend more and more time using and maintaining different tools and sifting through more and more results. To wisely manage development resources, teams must be able to effectively manage, filter, and prioritize all those issues.

To address these problems, development testing platforms have emerged to unify and manage all of this static analysis information in one place, simplifying the user experience and increasing visibility and efficiency at larger scales while providing relevant access controls and reporting. Development testing platforms are even starting to blur the line between static analysis and other types of analysis by utilizing – during the static analysis process – artifacts generated during earlier program runs. For example, these platforms can use code coverage information from test runs during static analysis to effectively identify missing test cases automatically. The traditional approach to this problem requires significant manual effort based on simple coverage thresholds. By leveraging data from different sources, these platforms are able to significantly reduce the manual effort and time required to accomplish this with other methods.

Selecting static analysis tools for Java

The most popular, free, static analysis tools for Java are probably Checkstyle, PMD, and FindBugs. While they all fall under the “static analysis” umbrella, their strengths are so sufficiently different that many consider the tools to be complementary rather than alternatives.

Checkstyle

Checkstyle is billed as “a development tool to help programmers write Java code that adheres to a coding standard[1],” although it does not strictly limit itself to coding standard enforcement. It provides a documented API for users to define their own custom checks. Typical coding standards utilize basic rules to make code more readable and reduce the likelihood that future code changes will introduce bugs. Standards tend to define conventions about formatting (white space, bracketing, naming, commenting, and so on), inheritance, and visibility. When adequately enforced, well-designed coding standards can help developers reduce risk. Enforcement can be difficult, though, since coding standards generate a lot of violations and there can be significant pressure to ignore noisy rules. With legacy code, this can make enforcing new coding standards unfeasible. While most of the issues identified by Checkstyle do not affect code correctness, robustness, or performance, there is real value in helping developers quickly understand code written by others. It is not always obvious how to quantify the risk represented by these violations and it is problematic to measure risk directly from violation counts, but changes in those counts can be a reasonable proxy for changes in risk.

PMD

PMD is described as “…a source code analyzer. It finds unused variables, empty catch blocks, unnecessary object creation, and so forth[2].” It, too, is evolving and the current checks focus mainly on syntactic oddities that might belie developer mistakes, such as overcomplicated expressions, empty blocks, unused variables, parameters, and class members. It also has a popular module to identify duplicated code. Because it is generally reporting “suspicious code” as opposed to specific coding errors or standards violations, the user will need to carefully select the checks enabled for everyday use. Because enforced rules are selected by the user, this tool can be useful for both legacy and greenfield projects, and it is often easy to correlate these counts with risk. Unfortunately, it might not be obvious whether reported issues should be considered defects or maintenance concerns.

FindBugs

FindBugs is probably the most popular of these tools. It looks for actual bugs in the code, as well as suspicious code and standards violations. Because of the wide range of reported issues, it is important to use a configuration that includes the most relevant checks for the project. This is especially true for legacy projects, as it’s easier to keep new projects clean from the beginning. Like PMD, any team can benefit from using FindBugs and associating issue counts to risk can be straightforward.

Commercial static analysis tools show similar diversity, identifying everything from standards violations to actual defects and security vulnerabilities. To illustrate how a commercial tool might compare to a free tool, I analyzed version 1.496 of the Jenkins job management system (www.jenkins-ci.org) using a proprietary static analysis solution and version 2.0.1 of FindBugs, with all checks enabled. On this code base, 852 unique issues were identified – with only 28 issues identified by both products. The proprietary solution found 197 unique issues, with 188 of those coming from high-impact categories (security and concurrency bugs, resource leaks, and unhandled exceptions like null dereferences). FindBugs found 627 unique issues, with 29 coming from those high-impact categories. In short, each of the tools found significant high-impact issues missed by the others, so using a proprietary solution or FindBugs alone will leave significant risk undetected.

Development testing – Tying it all together

Static analysis tools are a powerful ally in the software development effort for Java developers, as these tools enable developers to gain insight into risk throughout the software development life cycle. They are typically easy to automate, enabling users to spend their time fixing problems rather than running the tools.

When it comes to managing risk, more information is generally better – as long as that information illuminates actual sources of risk that developers care about. When deciding which tools to adopt, remember to consider not just the types of issues that analysis tools identify, but how those tools can work together to provide additional value. Also, be sure to configure them appropriately so that the number of issues doesn’t overwhelm your users.

Modern development testing platforms take testing tools to another level by unifying the data in one place, simplifying the user experience, and creating opportunities to provide even more value.

References

[1] http://checkstyle.sourceforge.net/

[2] http://pmd.sourceforge.net/

Jon Jarboe is Senior Technical Manager for Coverity, where he helps developers understand the value of adopting development testing and managing risk throughout the software development life cycle.

Coverity info@coverity.com www.coverity.com

Follow: Twitter Blog Facebook Google+ LinkedIn YouTube

Static analysis is a development testing activity with the potential to go far beyond simply checking code. When used as part of a policy-driven defect prevention strategy, static analysis can drive a software engineering team’s productivity and minimize fiscal, legal, and ethical risks associated with potentially faulty code. The reason more organizations do not realize the benefits of static analysis, however, is that it’s often homogeneously deployed as a tool for “finding bugs.” But the truth is that there are different implementations of static analysis that serve different purposes in the development process. And while it’s a foregone conclusion that software engineers should run static code analysis, the proper implementation of the right technologies is the difference between wasting time and money and reaching new software development heights.

Generally speaking, best practices are platform neutral – that’s why they’re called “best practices.” The subtleties endemic to embedded development notwithstanding, there are known standards for ensuring quality, regardless of platform. Avoiding memory leaks, for example, should be universal. Further, the relationship between static analysis and software isn’t necessarily defined by the application: It is defined by the purpose of the device. That said, running static analysis is a particularly important best practice for embedded software development.

Traditionally, embedded software is very costly and painful to access post-release. For this reason, most quality or validation activities are focused on eliminating the need to patch or refactor embedded code. Fixing errors post-release poses the greatest risk not only to the brand but also the bottom line. In some industries, particularly in the safety-critical realm, the consequences associated with software defects are so substantial that quality and verification tasks must be executed flawlessly. Software embedded into critical devices such as insulin pumps, weapon control systems, automotive braking systems, and so on, require a preventive strategy that uses a full range of static analysis capabilities; otherwise consequences could include costly litigation, C-level resignations, and even loss of life. This is opposed to agile, continuous development, Web-driven software applications such as smartphones, televisions, and so on, for which a preventive strategy is less important. To this end, the following discussion takes place on the preventive strategy side of the software development spectrum, examining various static analysis implementations:

• Integration-time static analysis
• Continuous Integration-Time (CI) static analysis
• Metrics analysis
• Edit-time static analysis
• Runtime static analysis

Integration-time static analysis

Running static analysis during integration to detect low-hanging fruit and egregious errors is a good starting point for implementing a preventive strategy. Integration-time static analysis simulates feasible application paths without actually executing the code, which is very helpful for systems in which runtime analysis isn’t possible. Static analysis can test across multiple functions and files and catch common memory problems, such as uninitialized memory, overflows, null pointers, and so on.

Static analysis serves a few purposes in terms of the development strategy when organizations begin with testing during integration. First, engineers can review the test results and determine how important they are for the particular application. Static analysis might uncover potential defects that might have a serious impact on software security, reliability, or performance. On the other hand, it could return things that the business might not care about. For example, business probably doesn’t care about a defect in a gaming console that causes the software to crash when an unlikely sequence of operations occurs. The user can simply reboot and continue enjoying their system. Resolving the same sort of issue in other contexts, however, might be crucial to preventing catastrophic consequences.

Static analysis can also help software engineers find potential defects that would have been very difficult to conceive of during the risk assessment phase. Engineers can catalog potential defects to improve future risk assessment iterations.

Continuous Integration-Time (CI) static analysis

After running integration-time static analysis, software engineers should have a stronger sense of potential systemic problems in the code. The next step is to run CI static analysis to enforce the coding policy outlined in the planning phase. This prevents the types of defects discovered during integration-time analysis.

For every issue discovered in static analysis, there are at least 10 more of the exact same thing in other places in the code. Static analysis is the ideal tool for addressing all violations of the same kind at the same time. This is opposed to chasing every possible path through the code. It’s far better to find the systemic problems in order to create an environment in which bugs cannot survive.

When we talk about static analysis, in many cases we mean anti-pattern analysis. A positive pattern is something that should be in the code. For example, a policy that requires engineers to use a typedef when declaring function pointers is a positive pattern static analysis rule[1]. This is in contrast to a policy that, for example, prohibits the use of the data() member function from a string class when interfacing with the standard C library[2].

Executing both types (positive- and anti-pattern) of static analysis is important, but it’s worth mentioning this distinction because if the organization spends the time to build a coding policy based on positive patterns, this ensures that software engineers are building code exactly how it should be per business objectives or compliance requirements.

Metrics analysis

Metrics analysis is a static analysis implementation that evaluates code characteristics and provides insight about the code that can help software engineers identify weaknesses (Figure 1). It is a critical sensor that can highlight areas of the application that can be prone to logical errors. Metrics analysis is an essential baseline measurement that should trigger further analysis, such as code review or some other remediation activity.

Metrics analysis is best used as early as possible because it might affect how software engineers write their code. Avoid trying to implement metrics analysis reactively or during the QA phase. The goal with metrics analysis isn’t just to detect potential defects; it’s to detect them in such a way that allows engineers to follow a sustainable coding trajectory. Run metrics analysis on potential defect hotspots, remediate any violations, and implement a pattern-based analysis rule to prevent future occurrences.

Any metric that correlates to potential problems is fair game. For example, a medical device company might use metrics analysis to measure cyclomatic complexity because a high score indicates that there are too many decision points for the device to handle during normal operation. Knowing that the complexity score exceeds the threshold set in the coding policy when there are 10 branches to cut, as opposed to finding out in the QA phase, will help keep the project on time and on budget. The organization might, for example, want to measure public variables because a high number might correlate to too many dependencies in the code. Each organization will need to decide which metrics can be correlated to possible defects in the code.

Edit-time static analysis

The static analysis sweet spot is while the developer is working in the editor. Running static analysis at edit time serves a few purposes. First, it points software engineers to potential problems. Second, it implements the risk assessment strategy by ensuring that any issues are remediated systemically.

But when should static analysis be implemented? We’ve discussed why implementing static analysis too late is a problem; however, it can also be implemented too early because there must be enough context for static analysis to provide meaningful information. Running static analysis on a character, line, or even statement creates too much noise to be useful. Enforcing positive design patterns ensures that new code is built as intended – while it’s being written. Running static analysis at edit time is a powerful way to promote the correct behaviors within the development team because feedback is rapid and in context of the code being written. Leveraging this type of analysis makes code reviews more productive because engineers should be able to correct policy-based errors immediately.

Runtime static analysis

Some static analysis patterns can detect defects at runtime. If the embedded target can accommodate the overhead, the organization should execute runtime static analysis to round out its preventive strategy. Runtime static analysis detects errors while the code is actually running, which enables software engineers to test real paths with real data.

Final note about static analysis and QA

In an ideal preventive strategy, errors found when QA runs static analysis should already be known and determined acceptable. This is because software engineers should have already tested against and adjusted design patterns to enforce coding policies. Violations at this stage mean that there is a problem with the process, such as improper static analysis rules. In these cases, QA needs to send the code back to development so they can find the systemic cause of the defect and implement a rule to prevent future occurrences. From this perspective, static analysis is a much better quality gate than a bug finder.

References

[1] Joint Strike Fighter, Air Vehicle, C++ Coding Standards, chapter 4.22 “Pointers & References”; AV Rule 176

[2] PCI Data Security Standard Version 1.2; “Requirement 6: Develop and maintain secure systems and applications”

Arthur Hicken is Evangelist at Parasoft and can be contacted at Arthur.Hicken@parasoft.com.

Wayne Ariola is VP of Strategy at Parasoft and can be contacted at Wayne.Ariola@parasoft.com.

Adam Trujillo is Technical Writer at Parasoft and can be contacted at Adam.Trujillo@parasoft.com.

Parasoft info@parasoft.com www.parasoft.com

Follow: Twitter Blog Facebook LinkedIn YouTube

Advanced static analysis tools are helping programmers say what they mean, and mean what they say.

Advanced static analysis tools are becoming a standard part of many professional programmers’ toolkits. At the same time, a growing emphasis is being placed on contract-based programming, where explicit preconditions, postconditions, and other contracts are added to source code to help enhance software safety and security as embedded systems grow more complex and interdependent. As these two trends meet, some interesting opportunities arise. In particular, certain advanced static analysis tools are beginning to directly recognize contracts, and some are going so far as to help the programmer create contracts, by inferring them from existing code. A review of advanced static analysis helps set the stage for a discussion of contract-based programming.

Reviewing advanced static analysis

Newer static analysis tools are no longer simply enforcing coding guidelines, but instead are delving into the semantics of program constructs, effectively simulating what might happen at runtime, to detect logic inconsistencies or security vulnerabilities. Typically based on compiler technology, these tools use advanced data flow analysis to determine where the program might go awry, by tracking the values that variables might have at runtime, and then checking whether those values are all properly handled by the program and whether possibly tainted data is properly vetted before being trusted. There are still challenges with such tools generating false positives, effectively false alarms, in places where the code is in fact safe and secure but the tool’s value tracking or taint tracking is inadequately precise. Nevertheless, improved automated error ranking algorithms and techniques, such as focusing only on differences between one analysis and the next, have made these tools valuable new weapons in the ongoing battle to improve safety and security at a reasonable expense.

Figure 1 illustrates how a static analyzer can use data flow analysis to track the possible values of a variable such as Count and determine whether any of these values might cause a problem at some later point. The values of a table are being displayed, followed by the average value. The classic “bug” here is to ignore the possibility that the table is empty, causing a possible division-by-zero error.

In this example, to avoid a divide by zero, the programmer has included an assertion that the table has at least one element (that is, “Table’Length >= 1)”. However, some data flow analysis is needed to verify that Float(Count) is non-zero in the division “Sum / Float(Count).” This requires the static analyzer to link the value of Float(Count) to the value of Count, the final value of Count to the number of loop iterations determined by Table’Range, and that number to Table’Length (X’Range means “X’First .. X’Last,” while X’Length means “(if X’First > X’Last then 0 else X’Last – X’First + 1)”). What is easy for the programmer can be more work for the static analyzer.

So what does the static analyzer do with “pragma Assert(Table’Length >= 1)”? Here is where analyzers differ, depending on whether they adopt a largely bottom-up or top-down strategy for finding errors that cross procedure boundaries, and how they integrate this with the notion of contract-based programming.

Where contract-based programming fits in

Contract-based programming is (among other things) the usage of preconditions and postconditions to express expectations about the inputs and outputs (respectively) of the functions and procedures (that is, the subprograms) that comprise the program.

In the example in Figure 1, the programmer’s intent is clearly that “Table’Length >= 1” acts as a precondition for the procedure. Unfortunately, this Assert is buried in the code for the procedure, rather than being readily visible to a caller. In languages such as Eiffel[1] or Ada 2012[2] where pre- and postconditions are part of the syntax, or in languages like C# or Java with extensions like Spec#[3] or Java Modeling Language (JML)[4], the programmer’s intent for the Table input to the Display_Table procedure can be expressed using an explicit precondition. For example, in Ada 2012, the specification for this procedure could be written as:

procedure Display_Table(Table: Float_Array) with Pre => Table’Length >= 1;

This specifies the aspect Pre, short for “precondition,” for the Display_Table procedure, so that it is visible to the caller and effectively becomes a contract on Display_Table, indicating that so long as Table is of length at least one, Display_Table can do its job correctly.

Static analysis: Checking and inferring contracts

Now back to pragma Assert in Figure 1. Without an explicit contract requiring the caller to ensure that Table’Length >= 1, the static analyzer could rightly complain since there is nothing preventing the caller from passing in a zero-length Table. However, many static analyzers use a different strategy. Rather than immediately complaining about the Assert, they rely on more global checks to determine whether or not there is a real problem, and only complain if there is a call that passes in a zero-length Table. As mentioned, these kinds of global, interprocedural checks can either be mostly bottom-up, or mostly top-down, as illustrated in Figure 2.

In a top-down strategy, the analyzer walks down from the entry point of the program, with actual parameters substituted in for formals at each call, until every call of each subprogram is identified, accumulating a set of possible actual values that is passed in for each formal. This value set is then used to determine whether the Assert might be violated via some particular chain of calls.

In the bottom-up strategy, analysis starts at the leaves of the program (subprograms that make no calls), analyzing each subprogram to determine which requirements it imposes on its inputs. In this example, the Assert(Table’Length >= 1) is converted effectively into an implicit precondition for the procedure. The static analyzer is essentially inferring the unstated contracts for each subprogram, which are then propagated to each call point, where the preconditions become implicit Asserts on the actual parameters at the point of call. This process continues up through higher-level subprograms, until ultimately the whole program has been analyzed.

The bottom-up approach can scale better than the top-down approach as programs grow large, but it depends on inferring potentially complex contracts, including conditional preconditions where the precondition for one input might depend on the value of another input. For example, for a procedure starting with “if X > 0 then Assert(Y > 0)” the inferred precondition should be “X > 0 ==> Y > 0”. Two advanced static analysis tools that infer contracts via a bottom-up analysis are the CodePeer tool from AdaCore[5], which analyzes Ada source code, and the Clousot tool from Microsoft Research[6], which analyzes .NET programs.

As explicit pre- and postconditions begin to appear in programs, using languages like Ada 2012, new synergies arise between these contracts and the capabilities of advanced static analysis tools. Explicit contracts can simplify interprocedural analysis, as the programmer has already done the hard work. The tool can simply check against the explicit precondition rather than having to propagate across the call. Within the subprogram, the tool can use the precondition as a precise description of the possible input values, with no need to guess the programmer’s intent.

Explicit contracts can also assist other programmers hoping to make use of the subprogram, since they act as machine-checkable comments and low-level requirements embedded directly in the code. But they only help if programmers write them. Since some advanced static analysis tools can infer contracts from the source code, they can offer to automatically insert them into the source. Tools like Clousot[6] allow the programmer to “bless” an inferred contract, having it become a permanent part of the source code.

The future: Proving while programming

The synergy between static analysis and contract-based programming may allow faster adoption of both technologies. As these two are integrated, a new approach to programming can emerge, where a programmer’s assistant is helping to infer and check contracts as the source code is created. Safety and security are being proved as the program is written, much as a spell-checker in a text editor can ensure that no misspelled words ever see the light of day. As these technologies mature, we can hope that insecure, unsafe programs will no longer be the norm, but rather safety and security will be built in from day one, with the bonus of machine-checkable, human-readable contracts accompanying code as it is written. Tools like CodePeer[5] and Clousot[6] are showing some of the possibilities.

References

[1] ECMA International, Standard ECMA-367, Eiffel: Analysis, Design and Programming Language, 2nd Edition, June 2006, www.ecma-international.org/publications/standards/Ecma-367.htm

[2] Ada 2012 Language Reference Manual, www.ada-auth.org/standards/ada12.html

[3] Microsoft Research, Spec#, http://research.microsoft.com/en-us/projects/specsharp/

[4] Patrice Chalin, Joseph R. Kiniry, Gary T. Leavens, and Erik Poll, “Beyond Assertions: Advanced Specification and Verification with JML and ESC/Java2,” Formal Methods for Components and Objects (FMCO) 2005, Revised Lectures, pages 342-363, Volume 4111 of LNCS, Springer Verlag, 2006, www.eecs.ucf.edu/%7Eleavens/JML/fmco.pdf

[5] AdaCore, CodePeer: Automated Code Review and Validation, www.adacore.com/codepeer

[6] Manuel Fähndrich and Francesco Logozzo, Clousot: Static Contract Checking with Abstract Interpretation, Microsoft Research, Redmond, WA, http://research.microsoft.com/pubs/138696/main.pdf

S. Tucker Taft is VP and Director of Language Research at AdaCore, a company focused on open-source tools to support the development of high-integrity software. He joined AdaCore in 2011 as part of a merger with SofCheck, which he had founded in 2002 to develop advanced static analysis technology. Prior to that Tucker was a Chief Scientist at Intermetrics, Inc. and its follow-ons for 22 years, where he led the design of Ada 95. Tucker received an A.B. Summa Cum Laude degree from Harvard University, where he has more recently taught compiler construction and programming language design.

AdaCore info@adacore.com www.adacore.com/codepeer/

Follow: Twitter Facebook LinkedIn YouTube

With the smorgasbord of Open Source Software (OSS) available for developers to dine from, it’s vital they “eat right” by choosing the OSS compatible with their existing project and IP needs.

Open Source Software (OSS) offers intelligent systems designers a veritable smorgasbord of tools and technology. Spanning the entire software stack, from boot code and drivers to OSs, executives to middleware, and application components to development tools, OSS provides readily available alternatives to both legacy commercial software and also to in-house code developed from scratch.

But dining at the open source table is not an embedded bean feast – code gathered à la carte might not always integrate easily to make a well-formed “meal.” While literally millions of OSS projects are available on popular forges and hubs, developers must take care to choose the right technology ingredients and tidbits to fit project and intellectual property needs.

The following examines resources and tools for discovering OSS projects and metrics for those projects. It also explores factors to consider when choosing OSS projects and components for embedded designs. And it serves up heuristic methods for choosing the OSS technology most appropriate to real-world embedded development needs.

Discovering embedded open source

Finding open source software is easy. Finding the right piece of OSS can be much harder. Luckily, options for finding and evaluating OSS are plentiful, and come in five categories: search engines, hosting sites, individual project sites, dedicated OSS discovery tools, and embedded platform distributions.

Search engines

Google, Yahoo!, Bing, Baidu, and other general-purpose search engines actually do an okay job at ferreting out OSS projects. A quick search on the string “open source embedded database,” for example, yields a rich mix of references and actual project sites and repositories. But while search engines are an okay starting point, using them can yield scattershot results.

Hosting sites, foundations

Another path is to go right to the source – the forges and hubs that host multiple projects. Until a few years ago, SourceForge would have been a developer’s prime destination, with its collection of 450,000 project repositories. But today, new projects are likely to find homes on GitHub (with 2.4M unique repositories), CodePlex (32,000 projects), Google Code (10,000 projects), Gitorious, and a long tail of other sites.

Yet another type of locale for project hosting is the gamut of open source foundation forges – the Apache Foundation, the Outercurve Foundation, the Eclipse Foundation, and others. These sites bring together usually related bodies of code (for example, IDE elements and plug-ins for Eclipse) and can boast several hundred hosted projects.

While repository aggregations and foundation sites are searchable by themselves, each still constitutes a distinct silo; however vast their portfolios may be, they don’t cover the entire universe of open source.

Project sites

Some projects eschew the crowded forges and build their own dedicated Web sites and repositories. These may be projects of broad community interest, of greater maturity, or merely the result of technical vanity. In any case, the main challenge is still finding the project, not in the relatively limited haystack of a forge but in the larger universe of the World Wide Web.

Discovery portals and tools – The Michelin Guide of OSS

Probably the shortest path to finding and also evaluating open source projects lies in portals that help developers discover, track, and compare open source code and the projects behind them. These free portals include Ohloh.net (owned by Olliance Consulting parent company Black Duck), Google Code Search, and others. These services track the full gamut of open source software, and like the projects they monitor, they are themselves open, letting users introduce new project repositories for cataloging and analysis.

OSS management platform tools also exist to help developers discover suitable homemade open source as well as “in the wild.” At companies with established policies for OSS use and deployment, developers can use these tools to peruse directories of vetted/approved open source code documented and/or maintained by their employers. These portfolios can also include code built and managed under the umbrella of “inner-source” and “corporate source” programs.

Embedded platform distributions – Prix fixe meals

If the organization has already committed to a prepackaged embedded platform distribution – a commercial or community-based Linux tool kit, an Android SDK, or equivalent – then engineers already have a library of applications, middleware, and utilities at their fingertips. Embedded distributions typically comprise 250 to 500 packages, with each package containing one or more unique, ready-to-use pieces of project code. Unlike downloading code directly from project sites, embedded distributions and SDKs usually include prebuilt versions of project code, tested and vetted for integration compatibility across packages. In many cases, these versions might not be the latest and greatest, and developers might need to turn to the original project sites to access the more current features and bug fixes. However, switching to newer versions of projects, while attractive, can break compatibility with other code in your stack, and also fall outside Service-Level Agreements (SLAs) from commercial suppliers.

Evaluating options, refining the OSS palate

Finding potentially useful code represents only half the challenge. Developers must also vet discovered code across a variety of parameters to determine if it is technically and legally viable. Factors to consider include code size, language, and quality; community history and dynamics; software licensing; and provenance.

Code size – Legacy embedded designs face severe constraints on code size. While tumbling DRAM and flash memory prices have made parsimonious provisioning a concern of the past, embedded software still benefits from compact code. Memory and storage eaten up by utility and infrastructure code are unavailable for differentiating software and for end-user content.

Because OSS starts with source code, the memory footprint of a given project or software component isn’t always obvious. Moreover, today’s device-based software stacks can contain ingredients cooked up in traditional compiled/assembled languages (C, C++, assembly), byte-code executed Java, and scripted/interpreted languages (PHP, Python, Lua, and so on).

The sites and tools mentioned earlier report both the language of projects and the Lines of Code (LoC) in each. If a project is truly size-sensitive, the best approach is to download and build the source code to determine actual binary size (or just examine the total size of scripted/interpreted code). Figure 1 uses Ohloh reports to compare source code growth in three database projects over time.

Language – Implementation language is as important as functionality and size. If a project is being developed in C, projects in Java or Python probably won’t integrate well into the existing or planned software stack.

Code quality – Code quality can prove rather difficult to gage. OSS discovery portals do report how well commented/documented OSS projects are. Other tools exist to vet the quality of code contained within a project, for example, open source Sonar and the popular Coverity suite.

Community dynamics – Important metrics of the health and quality of open source projects lie in the size and activity of the community behind it. Some hosting sites offer historical participation metrics, and some sites include contributor data and activity over the lifetime of a project. Figure 2 uses Ohloh reports to compare the waxing and waning of the developer community over time for three database projects.

Commit history – Tied to community dynamics is the commit history for a project – how often are changes committed to project repositories, over the project lifetime and for recent timeframes? In an immature project, change can appear to be fast and furious; for moribund projects, commits drop away to zero. Viable, stable, mature projects lie somewhere in between.

Licensing – Dealing with the diversity of open source license types and requirements is beyond the scope of this article. Of the 2,200+ recognized licenses, developers are most likely to encounter perhaps a dozen. (See a list of the top 20 open source licenses at http://osrc.blackducksoftware.com/data/licenses/; these account for 90 percent of all projects). The most important open source licenses are the GNU General Public Licenses (GPL, LGPL, AGLP), the Apache License (APL), the BSD license, the Mozilla Public License (MPL), the Eclipse Public License (EPL), and a handful of others. Learn more about these and others at the Open Source Initiative, OpenSource.org.

A larger challenge lies in reconciling project licensing with a company’s Intellectual Property Rights (IPR) governance and compliance programs. A related challenge is reconciling the requirements of different licenses for diverse code integrated into a single software stack.

Provenance – Knowing the actual origins of code can also help in finding support for code, as well as protecting the company from potential legal challenges. Many useful and important projects are associated with commercial organizations that help maintain the project and provide support for it. Most projects have a unified copyright (note: the Linux kernel does not), and many have established processes for determining provenance (for example, certificates of origin for code submission).

The choosy OSS diner

The goal here has been to serve code-hungry developers useful pointers for discovering, vetting, and ingesting open source software. The diversity of options and the surfeit of licenses need not require a particularly adventuresome technology palate – OSS is today truly mainstream and it is a rare embedded project that does not use and/or deploy open source software tools and components.

Matching the right OSS technology to your project is less like rocket science and more like pairing wines and food. More time “tasting” OSS will teach you where to look for compatible coding languages.

Bill Weinberg is Senior Director of Olliance Consulting, a division of Black Duck Software.

Olliance Consulting, a division of Black Duck Software info@blackducksoftware.com www.ohloh.net

Follow: Twitter Blog Facebook Google+ LinkedIn YouTube

Google’s Android mobile Operating System (OS) has quickly become the dominant platform for smartphone devices. Can Android be equally effective in the In-Vehicle Infotainment (IVI) sector? Despite Android’s many functional advantages, software developers must take a careful look at Android’s strengths and weaknesses before adopting the OS in modern IVI systems.

Developers today can make a strong case that Android is now the most successful portable OS of all time. In terms of recent smartphone sales, according to research firm IDC, Android devices represent a 68 percent share of the global market (quarter ended September 2012). This compares to a 17 percent market share held by Apple. Within 12 months it is expected that more than 1 billion Android devices will be in use – an achievable goal considering that nearly 700,000 new smartphones are activated every day. Easy access to software and development tools for Android means just about anyone from an individual engineer to the largest R&D department at a large corporation can get involved.

For corporations selling online services, it’s almost a limitation not to have an appropriate Android smartphone app. During the past five years, user expectations have migrated from seeing a good website to seeing a good mobile website to having an Android or iPhone app available. According to AOL Tech, the download rate for Android apps in 2012 is 1.5 billion installs per month, with a total of nearly 20 billion installs to date.

Unfair to compare

It was inevitable that individuals who own both a smartphone and a vehicle equipped with an IVI system would compare and contrast the two. The functionality of a typical infotainment system has evolved in the past 10 years, restricted by the lengthy development cycles of automakers and their traditionally conservative approach to product development. Quality and reliability are paramount, as well as the overwhelming need to keep costs low to ensure the final product stays competitive.

At the recent Paris Motor Show, several automakers announced their latest models embodying the concept of the always connected automobile. One such system was Renault’s Android-based R-Link infotainment system featuring built-in Android apps such as navigation, multimedia, and phone support via an online store for Renault-approved apps. Despite these and other IVI enhancements, any driver today can look at a contemporary smartphone and find much more functionality and personalization on that device compared to an IVI system. Carmakers are becoming increasingly desperate to incorporate this level of functionality and flexibility into a vehicle without compromising its safety and security. Using Android, there are several ways to accomplish this endeavor, each with its own set of advantages and disadvantages.

Bring Your Own Device (BYOD) to your vehicle

If the Android smartphone can be considered the ultimate infotainment device, then why not have it connected inside the car? This is the approach taken by the Car Connectivity Consortium, an industry alliance established to allow smartphone screens to be displayed on the infotainment head unit. Several infotainment platform providers, including Mentor Graphics, offer this approach, whereby the head unit acts as a thin-client display, with apps running directly on the smartphone. Connectivity is provided through USB cable today, but Wi-Fi connections are emerging. Bluetooth 3.0 may also offer sufficient bandwidth for video streaming between smartphones and IVI systems.

The advantage of this approach is that the phone connectivity technology will not become obsolete as the car ages, which is an important factor given that the typical smartphone enjoys a higher refresh rate over its lifetime. The concept of random IVI software updates is seen as too risky for the more permanent car-based system; OEMs want to keep the process under their strict control. Looking 10 years ahead, this means that the infotainment system can still be current and relevant because its functionality is based on the smartphone at the time.

This approach also presents a cost advantage, as the permanently fixed infotainment system costs less for an OEM or Tier 1 developer to design and maintain. Another benefit is related to shared or rented vehicles – the smartphone immediately personalizes the vehicle to which it is tethered, without needing to learn a new user interface every time. One example showing the advantages of integrating a smartphone into the infotainment system is Android car mode, which turns an Android phone into a better driving companion by providing quick access to key applications such as GPS navigation, voice-activated commands, and a phone’s contact list.

The main disadvantages of allowing smartphone screens to be displayed on the infotainment head unit are loss of control and marketability of the infotainment system as a car feature. High-end automobile manufacturers are now differentiating themselves through the sophistication of an infotainment system; they are not willing to pass that advantage over to phone makers. There is also the unknown security risk lurking in terms of the potential for someone to hack into the vehicle system via a smartphone.

Considerations for building in an Android OS

Many designed-in infotainment systems such as the Renault’s R-Link build Android directly into the vehicle and pre-load a number of approved and tested apps. This offers a pre-built, tested, state-of-the-art infotainment system to a prospective car buyer. The idea here is that it’s now possible for the vehicle owner to download additional Android apps from an online store managed by the manufacturer. The Android OS is kept isolated from other vehicle functions and apps are only offered from manufacturer-approved repositories to help protect the system from malware. However, as Generation Y Android users start to dominate the car buyer population, they will want the freedom to download their favorite apps and won’t be happy with a pre-defined mix decided for them.

Looking at this from an OEM perspective, adopting Android as a base OS poses a few significant business risks. Some OEMs are nervous about the omnipresence of Google as the owner and licensor of the Android OS platform. Because Google manages the release schedule and content of Android, many automotive strategists are wary about Android changes affecting their product release cycle. What would happen if the license or terms of use suddenly changed?

The original Android OS was designed exclusively for mobile smartphones, and it must be modified to handle the wide variety of audio streams in the vehicle with signals coming from reversing sensors, radio, DVD player, navigation, phone, and external sources. The middleware in Android that covers audio stream routing has proved difficult to modify and re-test; the intended infotainment system has to link in at several points including the audio flinger (mixer providing a single output at a specified sample rate), underlying audio hardware, and audio manager. Some developers are questioning why they should commit to the technology when it’s possible to dock a smartphone into the vehicle.

Embedded Android architectures

Developers can choose from several possible approaches to implement Android into a vehicle. Some vehicle manufacturers use Android as the core OS for the infotainment system, deeming that it’s secure and mature enough to fulfill this role. For designers who are not as bold and want to stick with Linux, Android can still be included as a guest OS in a “container” (see Figure 1). Using the Linux Container (LXC), resources can be assigned by the Linux host to the Android guest, which includes memory available for apps, access privileges, services available, and interaction with other domains. The container is intended to be a secure environment, so users can potentially download entrusted apps into this area.

Another technique for including Android in an IVI system is to use a hardware or software virtualization layer (see Figure 2). In this scenario, each OS or domain runs on a dedicated virtual machine, and the hardware resources available from the underlying host platform are shared. Communication is allowed in a controlled way between the different domains, and boot-ups may be independent, allowing safety-critical features running on a dedicated domain to be available more quickly than the infotainment or Android systems.

Several hardware platform providers provide isolated domains in hardware. Software virtualization is available using proprietary software from providers such as SYSGO, OpenSynergy, and Open Kernel Labs. These virtualization layers consume a small amount of overall resource (typically 1 percent to 4 percent) and allow a high degree of domain isolation and safety.

In a few years, all drivers will expect their vehicles to be permanently connected to the Internet. This will allow access to cloud data services, telematics, video and audio streaming, and apps download. It is no longer a question of if this happens, but rather when all of this becomes available to the general public. The explosion of Android in smartphones has ensured that Android apps will need to be accessible in vehicles, and users will decide if these are built in or accessed via a BYOD solution.

Andrew Patterson is business development director for Mentor Graphics Embedded Software Division, specializing in the automotive market.

Mentor Graphics Embedded Software Division Andrew_patterson@mentor.com www.mentor.com

Follow: @mentor_graphics Blog Linkedin

More sensitive commerce- and entertainment-related credentials assets are finding their way onto devices
with open operating systems, creating a wider array of attack vectors. The prevalent approach to mitigate
this threat is based on hardware isolation of an execution environment for sensitive information, ensuring
data privacy for end users and services.

Mobile devices and associated security standards and protocols have been around for many years. It is common practice to secure inbound and outbound communications and to protect the permanent data stored on a device. Standards and technologies that address these requirements are well-defined and have been widely adopted – for example, standards defined by the Internet Engineering Task Force such as Internet Protocol security (IPsec) and Transport Layer Security (TLS) or persistent storage encryption technologies for disk or flash such as BitLocker for Windows and dm-crypt for Linux. However, as lifestyles have become increasingly digital and smart devices have become an inseparable part of our daily activities, the need for security has increased significantly.

As technology has evolved and enabled additional use cases, more sensitive assets are finding their way onto devices such as commerce- and entertainment-related credentials. This development has increased the circle of potential hackers and created a wider array of attack vectors trying to exploit the vulnerabilities of modern embedded systems.

A recent prominent class of threats has resulted from the proliferation of “rich and open” Operating Systems (OSs) such as Android. While Android’s intentional openness – manifested as code and documentation availability, as well as a built-in debug tool – leads to ground-breaking innovations by a massive developer community, it comes at the price of innovative attack vectors. These vectors typically lead to a state where a black hat has managed to gain privileged execution rights and full access to all assets available to the Android OS at runtime.

The following discussion describes the prevalent approach to mitigate this threat or at least limit the scope of assets compromised and minimize the associated cost of such control loss on the rich OS. This approach, which is based on hardware isolation of an execution environment for sensitive information, is critical to ensure data privacy for end users and services, particularly in the case of video on demand, commerce, and banking services.

It is important to note that the phenomenon of devices exposing an ever-growing attack surface is not limited to the mobile consumer electronic space. Other embedded systems that were justly classified as “closed” in the past, such as automotive control units or industrial systems, are becoming vulnerable to a similar type of attack. The main code execution entities in these devices are no longer safe due to OS openness, along with a multitude of external connections.

An isolated execution environment

The key to this security approach is the introduction of a Trusted Execution Environment (TEE), as it is termed by the GlobalPlatform. This environment, depicted in Figure 1, is isolated from the feature-rich, performance-based Rich Execution Environment (REE). In a nutshell, the TEE is used to protect sensitive, software-based security services from the REE. Modern Systems-on-Chips (SoCs) typically base this isolation on one of the following hardware measures:

• A hardware-separated execution mode in the main SoC processor: This approach is evident in TrustZone technology developed by ARM and available on all members of the Cortex-A processor series. Note that in this approach, a CPU core is either executing code belonging to the TEE or the REE (typically switching back and forth between the two domains).
• A physically separated processing entity (a stand-alone CPU) dedicated to security tasks: This approach, which is more intuitively understood, though not necessarily better (as always, the devil – aka the black hat – is in the details), has been around for a longer period of time and is still used in many devices. Note that in this approach, the dedicated security CPU core is only executing TEE-related code.

A comparison of the two approaches is tightly coupled with specific implementation details, and the right method can differ from system to system. Another approach to achieve such isolation that is currently less prevalent in the embedded space is based on a hypervisor or Virtual Machine Monitor (VMM). With this method, execution environments or virtual machines are separated by a software layer (the hypervisor) that is running at a higher privilege level. Assuming a robust, mathematically proven implementation of the hypervisor, this software-based approach can be very effective.

An interesting term-related distinction in this context is parallel to the difference between a Trusted EE and a Secure EE. The term Trusted alludes to the basic assumption behind the usage model: Code that is placed in the TEE is trusted to be nonmalicious and bug-free. There is nothing inherent in the TEE that prevents ill-formed TEE code from revealing sensitive assets to the world outside the TEE’s “walled garden.” This trust implies the need for an exhaustive review and verification of code placed in the TEE, especially if this code is serving multiple unrelated security services running within the TEE.

TEE implementation

The TEE typically offers generic security services known as Secure OS, which includes handling secure boot, communications, persistent data storage, cryptography, secure platform management, and more. In addition, specific Trusted Applications (TAs) run in the TEE using these services. A TA is typically a component of an application of wider scope that runs in the REE and interfaces with the user and other parts of the OS.

An example of a usage scenario is one where the user tries to consume high-value content such as HD video obtained through a subscription service. The user interface and multimedia framework interface are performed in the rich OS while portions of the Digital Rights Management (DRM) scheme such as usage policy enforcement, content key extraction, and secure video path enforcement are handled in the TEE by a dedicated TA. This separation is required to comply with robustness rules published by the DRM scheme owner in an open environment.

A single-usage scenario can involve several TAs. Along the lines of the HD content consumption example, a user might want to send video from a handheld device to a remote and perhaps larger display using a standard such as the Wi-Fi Alliance’s Miracast. To comply with Miracast security-related requirements on an open system, the Miracast software stack must be located on the REE, communicating with a TA in the TEE implementing the High-bandwidth Digital Content Protection (HDCP) 2.x specification as required by the Miracast specification.

Careful attention must be given to the task of breaking down a use case into the parts benefiting from execution in the REE versus those mandated to be ported into the TEE (see Figure 2):

• Excessive code pushed into the TEE could lead to performance degradation (throughput and power consumption), as well as potential security issues. As stated earlier, the trust in the TEE code originates from a careful review of what is placed there, a task favoring minimal code.
• Not having all the parts needed in the TEE could increase the risk of potential security breaches, resulting in a greater probability of financial sanctions. Licensees of most, if not all, content protection schemes accept liabilities in the multimillion-dollar range. It is a common misconception that only the cryptographic parts of a security-related protocol must be well secured, while leaving the logic that connects the crypto outside of the TEE. In fact, tweaking the “harmless” glue logic completely circumvents the security scheme.

The complexity level rises significantly when multiple TAs are needed in the TEE. The basic requirement is for mutual distrust between those TAs (assuring that a compromised TA cannot compromise others). Mutual distrust is typically handled by the Secure OS security services layer. Nevertheless, there are scenarios where different TAs must collaborate and exchange information securely, such as the aforementioned content protection scenario employing DRM and HDCP link protection.

The road ahead

Notwithstanding its accompanying security benefit, the task of distributed embedded software development using TEE and REE is not trivial, and compared to traditional development actually hinders the development cycle. Being a relatively nascent technology, this finding is not surprising. The phenomenon could change once design teams become better acquainted with the concepts, related tools, and development environments. TEE usage adoption rates will be accelerated if service providers’ requirements mandate it.

One concern service providers have is the scalability of their systems due to the fragmentation of devices and their security capabilities. No one wants to redo an application repeatedly. This concern is currently being addressed by the GlobalPlatform standards organization. The GlobalPlatform plans to publish a formal process for TEEs entailing certification by accredited labs.

The openness of modern devices, along with the extended connectivity introduced by new use cases, calls for higher levels of security in the embedded space. Distributed software execution using an REE alongside a hardware-based TEE has the potential to form a robust security solution, meeting stringent requirements without compromising the user experience.

As with other technologies, the key for adoption is defragmentation through standardization, an ongoing process in which the concept of a TEE is gaining support. Stay tuned for more progress in TEE embedded software development.

Asaf Shen is VP of marketing and IP products at Discretix.

Discretix marketing-dx@discretix.com www.discretix.com

Follow: Twitter Blog Facebook Linkedin

Securing data is becoming more critical and more difficult to accomplish as embedded application development has increased in complexity, especially when different communication protocols are incorporated into embedded designs. Developers need to know the options for managing data in a secure way and understand the role of a database in maintaining security over data management channels.

Security is an important consideration when mobile devices and other embedded systems interoperate with other systems and components. Unauthorized access, eavesdropping, session hijacking, and other security threats can result in irreversible damages such as data loss, intellectual property theft, and malfunction.

Data management security is a fundamental requirement of applications developed for embedded systems. From industrial automation and medical devices to solar power inverters and even home entertainment systems, data must be protected both at rest on the device and during communication. But who bears responsibility for data security? All device components must employ a security-conscious design, from the application and embedded database down to the hardware.

Safeguarding embedded data

Embedded application development is becoming more complex, and developers are interested in learning how to manage data securely across all phases of development for embedded systems. Whether an engineer is building a mobile device, solar inverter, medical equipment, or any other embedded system, data security is the riskiest part of a design.

Authentication and encryption technologies are essential to secure data storage and distribution, but what does an embedded developer need to know to secure an embedded database? As securing data becomes more critical and regulators and consumers demand serious data protection, an application developer might ask: How do I ensure that my application will be secure? Should data be secured at the application level or at the database level? Can security be implemented by simply assembling the right combination of technologies?

As long as data remains local to an embedded system with no communication layer, security management is not very complex. However, as communication protocols such as TCP/IP are added to the design, security supervision becomes more problematic, and developers must learn about various options such as securing the socket layer so data can be accessed safely.

Securing data at the application level

Databases and applications can offer a safe haven to make data secure. Developers can encrypt data before it leaves the application and arrives in the database, but this is only viable for unsearchable data. For example, an embedded system that manages security for a gate will have a list of staff and their credentials, such as PINs or passwords. The credentials should be encrypted by the application so they can be verified individually. However, any information used to identify or list staff members must not be encrypted by the application.

Physical security is also important. A gate security system should not store data on removable flash media that could be easily replaced to circumvent security. However, even if data is stored internally, a dedicated attacker with physical access to the device can almost always access the data stored there. Storage-level encryption is necessary to protect sensitive data in this scenario.

Securing data at the database level

Encryption is a recognized security method where data is encoded with a specific encryption key and the same data can only be read by supplying the same key. File encryption is a way to keep the data secure, as it will block access to each database file until the application provides the correct key. This method protects data in case of media theft and, as long as access to data is limited to local connections, is a preferred method for offering security.

In the past, encrypting data before it left the system was a common way to manage and secure data. However, this approach can make it difficult to analyze data and search for individual information.

Securing remote access and data distribution

Database security is important to developers who are concerned with data confidentiality, integrity, and availability. While steps such as creating a procedure for end-user access can restrict physical access, database security requires special attention and greatly affects risk management for an embedded system. Developers of mission-critical applications and business intelligence systems experience critical safety vulnerabilities if malicious systems on the network or malware applications intercept access to confidential data.

How can a developer secure remote access from an unauthorized session? Remote access requires protection from unauthorized access, as well as eavesdropping and session hijacking. These faults can be caused by a lack of security for data management and data distribution.

When consumers access data remotely, they might connect to the database without authorization, allowing anyone to access this data online. Therefore, it is necessary to implement an authorization token so consumers can use passwords to access the database. This secures communications to prevent direct access to the database by an unauthorized party.

Embedded database security features

Some applications collect data locally and periodically post that data to a server on the Internet. Other computers on the Internet or local network can observe or tamper with that connection if it is not encrypted. Developers often look for security and authentication features in the embedded database to offer flexible data safety techniques that address these problems. Using database security features, developers can achieve data security in embedded applications by encrypting both network communications and storage media.

ITTIA DB SQL is a database software library for mobile devices and other embedded systems that offers secure file storage, remote access, and replication (see Figure 1). Whether a database file is only accessed locally or shared over a public TCP/IP network, the encryption features provided by ITTIA DB SQL ensure that data is protected from unauthorized access, eavesdropping, and session hijacking.

To protect data at rest on a device, each database file can be encrypted with an AES-128 or AES-256 key. Advanced Encryption Standard (AES) is a data encryption specification that has been adopted by the U.S. government and other governing bodies across the world. Even if the database is removed from the device, it cannot be read or modified without the encryption key. As a result, sensitive data can be stored on or backed up to the removable media on a consumer mobile device without compromising security.

Security becomes an even greater concern when an embedded device can share data with other devices and back-end systems. Whether data is shared over an active client/server connection or through passive replication, communications should be authenticated using a protocol such as Salted Challenge Response Authentication Mechanism (SCRAM) that does not require the database password to be transmitted over the network. This ensures that only authorized parties can initiate a connection and modify the embedded database.

Connections over a public network such as the Internet should also secure the communication channel with Transport Layer Security (TLS) or its predecessor, Secure Sockets Layer (SSL). This prevents eavesdropping from other devices on the network and man-in-the-middle attacks such as session hijacking that can compromise security even after an authenticated connection is established.

Whether an application targets Windows, Linux, Android, ThreadX, QNX, or one of the many other operating systems commonly found in embedded devices, software developers must consider the security implications of data sharing and storage. Selecting an embedded database that provides the required features is critical to the specification and design of secure applications. While it is ultimately up to the application to implement adequate security measures, an embedded database that offers the fundamentals for managing security is essential to protect embedded designs.

Sasan Montaseri is the founder of ITTIA.

ITTIA sasan@ittia.com www.ittia.com

Follow: Linkedin YouTube

Beyond the challenges of a tough economy and fierce competition, embedded developers face problems with deliberate and unintentional misuse of their software, competitive IP theft, and tampering. While implementing an effective copy protection and IP theft prevention plan can guard against security threats, it’s only the first step needed to unlock the potential value of source code. Organizations must understand how to use advanced licensing systems to protect against risk and better monetize their work, all while reducing operating costs.

Embedded developers are evolving. As embedded systems transition to become combinations of solution-specific software running on off-the-shelf hardware, the value of Intellectual Property (IP) has increasingly transitioned to software. This evolution marks a significant change in how these hardware-turned-software vendors must operate to maximize profitability and continue delivering a positive customer experience.

To successfully monetize their IP, intelligent device manufacturers need to leverage the four aspects of a successful software monetization strategy: control, packaging, management, and tracking (see Figure 1). Each aspect directly affects profitability by either helping reduce costs or increase revenue.

The reality is that today’s embedded device manufacturers are facing a much more complex business environment that goes well beyond providing protection from tampering and reverse engineering of their products. Software monetization strategies for intelligent device manufacturers must take into account how these core elements are connected. Considerations such as profitability, user experience, and usage control directly affect each other and should be approached comprehensively. When software monetization strategies are implemented successfully, the intelligent device manufacturer can offer a more efficient experience for the user and a more profitable solution to the market. It all begins with controlling IP.

The key to controlling IP

Intelligent device manufacturers put hours and hours of time and know-how into developing unique software code that meets the needs of their customers and drives their businesses. Controlling that IP is the foundation of software monetization, as intelligent device manufacturers face problems with deliberate and unintentional misuse of their software, product and feature overuse, competitive IP theft, product reverse engineering, and code tampering – all problems that have plagued traditional software organizations for years. The key to controlling access and use of a device involves controlling who is granted access to the software running the device, when they’re granted access, and to what extent.

Access control

The software embedded within an intelligent device is typically a vendor’s most valuable asset. It not only holds all the development secrets hackers or competitors would love to gain access to, it also determines how the product functions. Stolen code can end up in the hands of competitors or be used to reproduce knockoff versions of a similar product.

While theft of trade secrets is one threat, for many intelligent device manufacturers, so is tampering. Both scenarios have great potential to damage overall market share and therefore decrease revenue potential. Tampering with the software embedded within a device can change how the device functions. This can provide users with access to features they have not paid for or, even worse, result in regulatory compliance problems. Without proper protection, intelligent device vendors are unknowingly leaving their code vulnerable to tampering.

By effectively controlling access to software source code, intelligent device vendors can protect revenue and safeguard the integrity of their brands and products by preventing product tampering, reverse engineering, and IP theft.

Usage control

Usage control is the next piece of the monetization puzzle. Vendors must be able to control the use of their software at the product and feature level to prevent overuse of their offerings – deliberate or unintentional – and ensure that they are being fairly compensated. As the intelligent device market continues to mature, it will be critical for vendors to minimize manufacturing costs while achieving greater flexibility in their product packaging. This is accomplished through feature-based licensing. By providing customers the flexibility to license software features of intelligent devices already on premise, and by controlling access to that software, vendors can create new revenue opportunities.

Leveraging control to improve customer experience

An effective software licensing strategy will not only give vendors the means to control how their software is accessed and by whom, it will also provide them with a tool to help develop sophisticated packaging and pricing strategies. In addition to preventing unauthorized access, these systems lay the groundwork to change how the intelligent device industry is able to do business. Control over software at the feature level enables vendors to consolidate hardware stock-keeping units and provide remote upgrade and support services, in addition to opening the door to a whole new world of marketing and sales tools.

In the past, if a vendor wanted a premium and a standard version of a piece of equipment, they would build two applications for installation on two different hardware platforms. If a standard customer wanted to upgrade to a premium device, they would have to return their old device and wait for the vendor to ship them a new one.

With feature-based licensing and entitlement management, device manufacturers can develop and maintain a feature-rich application installed on all devices. The functionality of the device is controlled through licensing. This enables software vendors to ship the same product with different functionality to different customers at varying price points and upgrade products remotely with lower support and fulfillment costs, thus delivering a better customer experience.

Tracking usage to evolve business strategies

The benefits of software licensing do not stop with control or packaging enhancements. Implementing a sophisticated licensing and entitlement management system also provides a means to to track product activation and usage right down to the feature level. Intelligent device vendors can use this information to drive decision-making around product packaging, roadmap investment, sales, and marketing strategies. Product management and engineering teams can discontinue feature combinations that are unpopular and create software packages containing the most valued features that customers want.

Marketing and sales teams can utilize customized reports to determine what, when, and how products are being used and leverage this data to plan, launch, and execute more effective sales and marketing activities (see Figure 2). End-user registration data can also help vendors who sell via multiple channels to identify and gain direct access to every individual who uses one of their products.

Licensing best practices: case studies

Because most intelligent devices, software and licensing systems, end users, and ecosystems are all unique, examples of licensing best practices can help illustrate some key elements of an effective software monetization strategy.

Example 1: Protecting software code from theft and resale

The first example is a software publisher who provides a compression algorithm that is nearly lossless. It is imperative that the algorithm not be deciphered because it is critical IP and unique in this company’s industry. The software publisher leveraged the code-wrapping feature of a software licensing solution to protect their code from reverse engineering and therefore protect their competitive IP from getting into the hands of pirates or the competition.

Example 2: Leveraging feature-based software licensing

Another example is one of the largest networking companies in the world that decided to monetize the software that ships on their appliances. The company used a sophisticated software licensing and entitlement management implementation to protect their code and created smart, feature-based licensing packages for their enterprise customers to maximize the return on their IP.

Example 3: Protecting software from tampering

The last example is a company in the manufacturing industry that develops machines that create end-to-end packaging of consumer food products such as milk and orange juice. The software that runs these machines is programmed to comply with dozens of public health and safety regulations. The company’s IP protection concerns centered around controlling access to the software running the machines and the ability to tamper with key parameters that control processes such as pasteurization. This company used a software monetization solution to protect the software from being accessed and control who can change the parameters that control the machines.

Confidence in greater protection

Shifting from an equipment manufacturer business model to that of a software company does not happen overnight and typically occurs in phases. Intelligent device manufacturers who embrace the transition and use software monetization tactics to overcome the challenges they face will be able to vigorously pursue greater market share and reduce manufacturing and inventory costs with the confidence that they are protected against competitive threats to their IP. They will also be able to more cost-effectively expand their product lines and bring innovative devices to market. In short, those vendors making the transition to a software business model will be more nimble and better positioned for the future.

Michelle Nerlinger is director of product marketing for SafeNet’s software monetization solutions.

SafeNet sentinelquestions@safenet-inc.com www.safenet-inc.com/software-monetization-solutions

Follow: TwitterLicensing TwitterNerlinger Blog Facebook Google+ Linkedin YouTube

Virtualization means different things to users with different types of applications. Most forms of virtualization employed in IT server environments aren’t of interest to embedded system developers because they don’t ensure that processing of time-critical tasks is deterministic. Instead, the way for single and multiprocessor platforms to support multiple operating environments while maintaining real-time responsiveness is to functionally partition processor resources so that they are controlled by specific operating environments, which run directly on the processor silicon rather than on virtual machine implementations.

The origin of embedded virtualization technology came about with the idea of creating an environment where a Real-Time Operating System (RTOS) could work alongside a General-Purpose Operating System (GPOS) such as Microsoft Windows. Embedded virtualization creates a partitioned environment in which the two OSs and the applications on them run on a single platform as if they were running on two separate platforms. The advantages of doing this are clear: system cost and complexity can be reduced if fewer processing platforms are required to serve the application’s computing needs. Product reliability can be enhanced as well if systems can be built with fewer hardware elements.

Back in the early ’80s, machine builders saw the opportunity to leverage the PC platform to build control systems for their machines. The first such applications were relatively simple, and the focus was mainly to leverage available hardware that was substantially lower in cost than specialized control hardware.

As the PC evolved with the addition of Windows, numerous application software packages were introduced, driving a new standard of Human Machine Interface (HMI) with supporting graphic engines and software tools. Machine builders saw the opportunity to use Windows to create advanced HMIs that could simplify their machines’ setup, operation, and maintenance. However, Windows-based PCs could not be used for portions of an application involving time-critical control because Windows, by itself, isn’t an RTOS and is not capable of performing control functions with determinism. Hence, embedded system designers would typically add a real-time computer subsystem to the machine in addition to the PC to deliver a full suite of product functionality.

RTOS suppliers, on the other hand, have not had the resources to build the kind of graphic software tools and support that are available for Windows. A few saw the opportunity to couple their OSs to Windows in order to add RTOS functionality to Windows-based systems on a single computing platform.

The benefits of combining an RTOS with Windows on one machine were obvious for embedded systems OEMs, but the technical issues associated with doing that were very complex. Running two OSs on one computer wasn’t a new concept; it had been done 10 years before on mainframes with virtualization technology. That technology virtualized the whole computer platform, essentially creating an interface layer between the OSs and the hardware much like modern server virtualization technology does today.

The fundamental problem with this is that isolating the OS and application software from direct access to the hardware causes nondeterministic time delays when the application software needs to interact with its I/O. Real-time applications, however, must have direct access (sometimes called bare-metal access) to the devices that the applications need to control, so that the software can write or read data to and from the I/O devices in a timely, deterministic manner.

Embedded virtualization solves the multi-OS determinism problem

A method must be devised to partition the platform resources so that the RTOS can gain direct access to I/O and interrupts that are necessary for it to run an application deterministically. GPOSs like Windows do not allow a co-resident RTOS to control its I/O devices.

Instead, GPOSs typically take control of all available I/O on the platform during installation. Barring the option of modifying Windows, which would bring a whole new dimension of problems, a means of reserving I/O from Windows had to be devised. And since this was initially done back in the days of single-core processors (Figure 1), techniques had to be developed for the processor to switch context from the RTOS to the GPOS with minimum overhead. These are the principles of embedded virtualization, principles that have been validated in thousands of successful embedded system products.

Interprocess communication enables task coordination

Multiple OSs running applications in shared but separate environments create the need for applications to pass data to each other. This could easily be performed with a simple block of reserved memory, but would require some level of housekeeping by the applications and would be cumbersome to manage in real-time systems, where messages need to be delivered and read at particular times.

The communication process needs to be structured in such a way that message delivery occurs when expected to maintain determinism. Communication has to take into consideration the priority of the message with respect to the priority of other real-time tasks that are running at the time, so that the message will be delivered at the right time or in the right sequence. This is particularly important when the communication is between an application running in a non-real-time GPOS environment and an application running on an RTOS in a real-time environment. An unprioritized event must not be allowed to interrupt a prioritized task.

Multicore processors aid functional partitioning

The introduction of multicore processors caused some of the rules to change. In principle, processors no longer need to be shared. Each OS can have its own processor core (or multiple cores can be dedicated to a single OS); however, in practice, OSs such as Windows assume that all processor cores belong to them at installation.

Setting up a multicore system so that the GPOS is in control of some cores and not in control of others requires a way to tell the GPOS which cores are not available to it. With the proper means of resolving this, a four-core processor could support several configurations of GPOS:RTOS, including 3:1; 2:2, and 1:3 (see Figure 2). This flexibility allows the user to optimize the platform’s computing resources depending on the application’s requirement. Whereas an application with a complex Windows portion and light real-time requirements could be configured with three cores running the GPOS application and one core running the RTOS, an application with multiple real-time control functions running simultaneously and communicating with a simple HMI might have one core dedicated to Windows and three to multiple instances of the RTOS.

With the possibility of running several RTOSs at the same time on a multicore processor, the communication system that was initially developed to communicate between the GPOS and the RTOS on a single shared processor can be extended to enable communication between multiple instances of the RTOS and GPOS. In theory the system architecture could be expanded to create a network of OSs talking to one another, each running application elements that are particularly suited to its own environment. As with the I/O needs of the system, the communications structure needs to maintain and support the real-time determinism requirements of the real-time subsystems.

Virtualization enables deterministic communication across platforms

One embedded virtualization environment that has proven itself in mission-critical applications is the INtime RTOS family from TenAsys Corporation. INtime’s embedded virtualization technology encapsulates the principle of partitioning the PC platform to enable Windows and the RTOS to run side by side. INtime facilitates deterministic communications between instances of the RTOS and Windows with a global object networking system called GOBSnet. This consists of a built-in communication network that allows multiple applications on separate OSs to communicate at the process level in a deterministic way.

Using Ethernet with an addressing scheme akin to that of a URL, GOBSnet was extended to enable separate system functional blocks, called nodes, to communicate deterministically with each other on the same multicore processor or across platforms that are physically distinct. In this manner, large and complex applications can be distributed across several nodes (see Figure 3), simplifying their creation, debugging, and optimization while leveraging the parallel processing capability of multicore processors. This allows OEMs to produce a range of products at different cost points or functionality levels by scaling the number of processors or processor cores that are employed.

Embedded virtualization techniques have been developed over more than a decade of use in real-time applications, but the full potential of these methods to revolutionize embedded system design is just now becoming clear with the advent of processors that include increasing numbers of CPU cores. Along with global object networking support, embedded virtualization is primed to become the standard way of building large multi-OS systems.

Chris Grujon is marketing director for TenAsys Corporation.

TenAsys Corporation Chris.Grujon@tenasys.com www.tenasys.com

Follow: LinkedIn

Along with multicore technology, virtualization software has become invaluable to developers seeking to combine several embedded functions in a single hardware platform that boosts system performance and decreases development costs. Warren describes the critical functions of hypervisors and processors with hardware-assist features and presents a few examples of platforms that show how virtualization can consolidate dissimilar functions while maintaining isolation and security.

Virtualization is rapidly becoming one of the hottest technologies in the embedded space, offering designers a host of new hardware and software options for product development and future modifications. With the proper architecture, virtualization can be used to combine multiple embedded functions into a single hardware platform to minimize development costs, power requirements, and the number of system components. This consolidation feature allows designers to merge existing applications with diverse operating software into a single system without the need to modify legacy code.

Combined with the recent popularity of multicore technology, virtualization can also boost the performance and responsiveness of individual software segments by assigning additional processing power. Similarly, virtualization allows General-Purpose Operating Systems (GPOSs) such as Windows or Linux to be easily combined with real-time software or safety/security-critical functions while retaining the required determinism and isolation.

Originally introduced by IBM in the 1960s for enterprise servers, virtualization enables multiple copies of the OS to run in parallel on a single CPU, thereby reducing the number of machines required. Unlike the enterprise environment where hardware and operating software are consistent across platforms, the embedded industry employs a wide variety of processor architectures and I/O structures, so virtualization cannot be applied the same way. For example, enterprise-level applications typically create virtual copies that represent the entire machine environment to maximize CPU utilization. Unfortunately, this comes at the expense of responsiveness to external events, making this approach impractical for time-critical applications.

The latest virtualization software now available for embedded applications allows the development team to independently allocate system resources including memory, additional processors, and I/O to each operating environment to optimize performance.

Hardware allocation

Virtualization platforms are built by adding a real-time Virtual Machine Monitor (VMM) or hypervisor software layer directly above the hardware to create and manage individual partitions that contain guest OSs. The hypervisor allocates system hardware resources such as memory, I/O, and processor cores to each partition while maintaining the necessary separation between operating environments.

A critical function of the hypervisor is to maintain isolation between partitions and continue running even if another OS crashes. Multicore processors allow hypervisors to create a variety of configurations to support embedded development. For example, an OS can run on a single core or be spread across multiple cores to increase performance. Similarly, multiple OSs can also run on a single core if timing is not an issue.

Several variations of hypervisor software are available for virtualization applications. Full virtualization is a nearly complete simulation of the actual hardware, which allows a guest OS to run without modification. Partial virtualization simulates some but not the entire target environment, so guest software might need some modifications to run in this environment. Using paravirtualization, guest programs are executed in their own isolated domains without a simulated hardware environment. Although guest programs must be specifically modified to run in a paravirtualization environment, having the guest OS communicate directly with the hypervisor can improve performance and efficiency.

The latest generations of embedded processors include built-in hardware functions to increase performance and speed up interaction between virtual environments. For example, Intel Virtualization Technology (Intel VT) includes facilities to trap certain VMM instructions in hardware and simplify the hypervisor functions to reduce virtualization overhead. Intel VT for Directed I/O adds hardware accelerators that allow secure assignment of specific I/O devices to specific OSs to decrease the load on the processor and accelerate data movement. For example, a hardware-based network controller can be used to offload the Ethernet stack processing to improve the performance of high-speed networks.

Another improvement is to implement I/O queuing mechanisms so that operating software does not waste time waiting for operations to finish. In addition, specialized Intel functions such as Extended Page Table and Page Attribute Table provide a hardware-assist to the partitioning and allocation of physical memory among virtual machines.

Virtual platform examples

Software vendors offer designers a variety of hypervisor-based products to capture the advantages of virtualization for embedded systems. For example, PikeOS from SYSGO incorporates paravirtualization technology to create a combination Real-Time OS (RTOS) and virtualization environment that enables multiple OS partitions to work on separate sets of resources within a single machine (see Figure 1).

The recently released PikeOS version 3.3 supports a wide range of operating software including Linux, ARINC 653, POSIX, Android, and others. PikeOS also runs on multiple single- and multicore processor architectures such as x86, PowerPC, MIPS, ARM, and SPARC/LEON. Multicore processor support offers flexibility to users who can select an execution model ranging from pure Asymmetric Multi-Processing (AMP) to full Symmetric Multi-Processing (SMP). PikeOS is certifiable to safety standards such as DO-178B/C, IEC 61508, EN 50128, and ISO 26262. The PikeOS microkernel architecture is small and compact, resulting in real-time performance that competes with conventional proprietary RTOS products.

Virtual platforms that combine safety-critical embedded functions with a large GPOS must contain security provisions allowing unaffected partitions to continue operating in the event of a software failure or cyber attack. The recently released LynxSecure version 5.1 hypervisor from LynuxWorks offers military-grade protection features for customers building secure embedded systems. LynxSecure 5.1 provides two types of device virtualization including direct assignment of physical devices to individual guest OSs for maximum security and secure device sharing across selected guests for maximum functionality (see Figure 2).

LynxSecure also offers two virtualization schemes: para-virtualized guest OSs such as Linux for maximum performance and fully virtualized guests such as Windows, Solaris, Chromium, LynxOS-178, and LynxOS-SE, requiring no changes to the software. Another key performance feature LynxSecure delivers is the ability to run both fully virtualized and paravirtualized guest OSs with SMP capabilities across multiple cores.

These products demonstrate how virtualization technology allows designers to consolidate dissimilar functions while maintaining the required isolation and security. Along with a multitude of new software offerings, companies selling off-the-shelf boards and modules are now implementing hardware configurations that are friendly to virtualization applications. These boards have onboard memory that is easily configured for virtualization, along with smaller form factors and lower power requirements to support consolidated systems.

All of these products and design advantages point to a long-term, continuing trend in virtual technology for the embedded marketplace. Although it might require a change in embedded design philosophy, virtualization technology has developed into a valuable weapon in the developer’s toolkit.

Nobody likes a complainer, especially when you have to live with them day-in and day-out, and it seems that modern compilers spend a lot of time complaining about your code. Joe discusses how to live harmoniously with your compiler and generate the best results in the shortest possible time.

ECD: How can designers give themselves the best chance of having their compiler properly realize the intent of their source code?

DRZEWIECKI: I’ve met a lot of engineers who thought that compiler error messages and warnings were criticisms of their coding style or practice. Now that I’ve been working with compiler developers for a while, I can tell you that an error or warning is most probably because the compiler doesn’t understand the source code that you’ve written. Now, that should strike you immediately as a big red flag, ensuring that what you wanted to happen is probably not going to happen with the code the compiler generates. So, turn on the errors and warnings and deal with them.

ECD: Even good coders can routinely get the compiler to generate thousands of errors and warnings. Are you telling me that they have to wade through that mess and fix them all?

DRZEWIECKI: Only if you want your code to work! Every single one of those messages is the compiler telling you that it’s not sure how to interpret what you’ve written, or worse, that the compiler is pretty sure that what you said is not what you meant. As miserable as cleaning up compiler errors and warnings is, I guarantee that finding the misbehaving code on your board is 10x worse. I’m sure many developers have found code that they meant to work one way and it worked another – usually after painstaking hours of debugging, late nights, and the boss breathing down your neck. That’s no fun!

There are a couple of techniques that you can use to deal with the whiny compiler. First, write small chunks of code and compile them frequently to make sure the compiler understands them. Some developers may be saying, “Yeah, yeah, I’ve heard it all before, but I’m late, and …” It’s your choice. Like the old oil-filter commercial said, “You can pay me now or pay me later.” If you think you’re late now, just wait.

The second tip is to start at the very beginning, clean up the first error or so, and then recompile. I’ve seen instances where a single missing semicolon can generate so many error messages that the compiler stops printing them. Fixing that one missing semicolon cleared up a thousand errors. It’s not as bad as it seems, and turning off compiler errors is the epitome of being penny wise and pound foolish with your time (see Figure 1).

ECD: Is there anything else that can be done, besides ensuring that the compiler can properly interpret the source code?

DRZEWIECKI: Almost everyone knows what they should do when coding – short functions with a single purpose, clean control flow, good encapsulation of data (no global variables), and limited pointer/array usage. If you have a tough time being disciplined in what you know would make your code work better, I strongly suggest using a Motor Industry Software Reliability Association (MISRA) checker. MISRA has a number of rules that limit the most often abused features of the C programming language. If you stick with MISRA, you’ll get more deterministic execution and much less opportunity for code that gets “lost in space.” You may even be surprised to see your code size shrink after you’ve coded things so cleanly.

ECD: Even with clean coding and following best practices, the compiled code often still won’t fit in the designer’s chosen device. This often leads to the use of optimizations, which can break the code. Is there any way to avoid creating problems with the code when using optimizations?

DRZEWIECKI: A compiler’s most important responsibility is to generate “correct” code based on the source you give it. You know what you wrote. The compiler knows what it read. Since they’re the same thing, how can they be different? One of several answers lies in the mystery of code that has “no effect.” When the compiler reduces the size of your code, it can’t arbitrarily throw things away, so one of the tricks the compiler uses is to look for code that has “no effect.” It’s important that you understand what a compiler means by this. You may have a timing loop that looks like this:

Delay_ms(int NumberOfMilliseconds)

{

 long long i;

 for(i = 0; i < 5280

On the surface, open-source software tools appear to be a free solution for today’s embedded application developers. Just download the software and you’re ready to go, with the entire open-source community there for support. Yet, as is often the case with free software, it’s not quite that simple. Developers must uncover the hidden costs and limitations of open-source software and find alternatives to using it.

To an embedded application developer, open-source software seems like a fantastic option, whether it is used as the basis for an application or if open-source tools are used to accelerate the development process and speed time to market. Open-source software is, by its very nature, free – meaning without cost or payment. In an environment where managers are constantly trying to limit the costs around development, open source sounds like a great idea, and depending on each developer’s unique situation, it could be just the solution needed to address budget constraints.

Consider a few of the pros. First, open source is easy to acquire. With open source, developers just download and use the software. Code quality for open source can also be a positive. If the code is already decent, it will only improve as the community improves upon it. Leveraging the open-source community for help, support, and feature development is one of the greatest assets of open source. The feeling that “we’re all in this together” helps all boats rise.

Taken together, these benefits can make a developer wonder if a downside to open source exists. As it turns out, open source does have hidden costs that only become apparent once an organization is in the thick of the development process. While open-source products may be the right fit for some developers, others using open source can experience a variety of pitfalls after it’s too late to consider alternatives.

Let’s review open source’s impact on four key parts of the development process – using open-source code as the basis for developing embedded applications, customizing open-source platforms to accelerate builds, open-source sprawl and visibility, and support for time-sensitive issues.

Customizing open source for embedded environments

When considering open source to serve as the basis for a network infrastructure platform, set-top box platform, medical device, or any other embedded application, consider the amount of customization required to address the specific industry’s needs.

When implementing a new open-source project, a team either needs to customize the software to fit the needs of their environment using valuable resources from their own team or use funds to bring consultants in to do the same thing. The consultant route can save time that would have been spent customizing the software, but it still requires developers to spend time managing the process.

Maintaining open-source software is another time sink. Nine times out of ten, scripting and fixes are doable but come at the cost of developer hours that could be spent more wisely on developing the product, not just fixing open-source software. With the high costs of developers, that could represent a significant portion of development resources allocated to the “free” solution.

Optimizing open source to accelerate builds

Developers need to optimize their builds in a way that maximizes the productivity of the engineering organization. This means developers don’t sit around waiting for their individual builds to finish – a process that can typically take minutes or hours at best and days at worst. Build accelerator tools on the market today speed builds primarily through a process known as dependency-aware parallelism distributed across large-build clouds as well as clever caching and avoidance techniques, enabling fast and reliable incremental and full builds. Some large development organizations have optimized data centers employing these accelerators to speed builds for distributed global teams.

Open-source options are available for managing and accelerating software builds, but they entail clear limitations and caveats. For example, when using common software construction tools such as GNU Make and SCons, it’s typically costly to build in the necessary scalability and reliability to support and accelerate software builds at the appropriate level. This problem multiplies as organizations scale up to more developers, more projects, and a greater need for faster feedback loops throughout the development life cycle.

Another common problem with open-source build tools is the lack of visibility and “debugability” of the inner details of the software build structure, resulting in costly manual maintenance and long lead times for an organization looking to become more efficient and agile. The visibility to go back and identify what went wrong once the build failed simply isn’t there. As multiple teams combine their efforts to bring a product to market, this lack of visibility across teams slows down troubleshooting and therefore the ability to speed time to market. In an industry where time to market is everything and, increasingly, multiple releases and fixes for the embedded application are required, this level of support simply isn’t acceptable.

When considering alternatives to open-source tools, teams should ensure that acceleration tools don’t lead to broken builds, can leverage their existing hardware resources for parallelization, and can work with their existing toolsets and processes. 

One such alternative is Electric Cloud’s ElectricAccelerator (Figure 1), which executes parallel builds on a single machine or across a cluster of standard servers, reducing full or incremental build times by up to 20x. The key to this acceleration lies in patented dependency management technology, which detects and manages dependencies at the file level to ensure accurate builds. ElectricAccelerator plugs into existing build and release architectures, without the need to modify existing build scripts and tools. As an add-on to the build tool, ElectricInsight provides an intuitive, graphical representation depicting how the builds are structured and run, giving build managers the ability to pinpoint performance problems or conflicts across all of their builds. Instead of manually pouring over thousands of lines of build output files, error detection and performance tuning can be done in seconds.

Open-source sprawl and visibility

Because open-source platforms are easy to acquire, most departments end up implementing their own individual versions, often customized to meet their specific needs. For larger organizations, this can mean dozens of instances of continuous integration tools implemented within the same development organization, creating an unmanaged, siloed environment, or in other words, an open-source sprawl.

The resulting costs from such sprawls can be significant. First and foremost, this is not a model that scales easily. Second, management has little to no visibility as to the progress of the development effort. With a lack of standardization, there’s no way to truly assess progress or troubleshoot problems across interdependent projects. From a management standpoint, this is frustrating because efforts are uncoordinated and requests for costly resources come from all directions.

Open source and support

When using open source, developers always face the question of support. When time-critical issues must be resolved, the questions may need to be directed to the community because there is no official support system. Mission-critical applications that affect an organization’s revenues must be solved in a time-sensitive manner in collaboration with professionals who can be held accountable. There is something to be said for having someone on call on to fix a problem instead of relying on a community.

Taken together, all of these issues can easily compound to introduce significant issues in any embedded development process. That’s not to say open source can’t be a valuable tool for the embedded developer – it most certainly can. However, the cost of using open source for development efforts comes with limitations and requires time and effort for which organizations must budget. So when developers decide to depend on open-source technologies, they should make sure they’re aware of the true price of “free.”

Ashish Kuthiala is head of product marketing at Electric Cloud.

Electric Cloud kuthiala@electric-cloud.com www.electric-cloud.com

Follow: Twitter Facebook Google+ LinkedIn

Follow: Ashish Kuthiala LinkedIn

IDC is predicting that 15 billion intelligent devices will be connected to the Internet by 2015. This explosion in connected embedded devices has spawned a new generation of hackers targeting mobile devices, automobiles, medical equipment, and other systems. Alan discusses what these latest security threats to embedded devices look like and what steps companies should take to protect their devices from attacks launched via the Internet.

ECD: What are some common threats and attacks against connected embedded devices?

GRAU: We are seeing a surge in attacks against embedded devices. Attacks range from simple automated probes to sophisticated attacks targeting specific features of the embedded devices.

IP and Web attacks that have long been used against enterprise networks and Web servers are now being used to attack embedded devices. Hackers have compromised medical devices, reprogrammed printers, and even hacked antitheft and vehicle control systems in cars. The list of possible attacks is limited only by the creativity of hackers.

A few other common threats are dictionary attacks, where hackers attempt to log in and gain control of the embedded device using weak or default passwords, and insider attacks, where disgruntled employees steal passwords and sell them to hackers.

ECD: What steps can designers take to protect their devices from these attacks?

GRAU: Security needs to be considered from the very beginning of the design phase. Engineers must assess the possible attack vectors available to hackers. Each interface provided by the device is a potential attack vector for hackers. Wi-Fi, Ethernet, Bluetooth, serial communication, and even debug ports have been targeted by hackers. Once the risks are determined, engineers can begin designing security measures for the identified risks.

Many embedded devices include security protocols such as Secure Shell (SSH) or Secure Socket Layer (SSL) to ensure secure communication with the device. While that is an important step, it is not sufficient. A firewall is the critical layer of security that is missing in most embedded devices. A firewall allows the creation of policies that define and enforce what communication is allowed with the device. The policies define, at a minimum, with whom the device communicates, which protocols are supported, and which ports are open. An embedded firewall is integrated into the communication stack and blocks packets at the lowest layers of the stack. By enforcing communication policies, many attacks are blocked before a connection is even established.

Consider a Supervisory Control and Data Acquisition (SCADA) controller that incorporates the Icon Labs Floodgate firewall and is configured with communication policies that define a set of trusted senders and block all ports and protocols not used by the device (see Figure 1). If hackers attack the device, they will be blocked because the communication is not originating from a trusted sender. Even if hackers steal passwords from an insider, they will not be able to log in to the device because they are not trusted senders. The firewall will block packets at the IP layer before a log-in is attempted.

ECD: How difficult is it to port security software to an embedded system? What are the impacts on performance and memory size?

GRAU: Most embedded systems require security software that is designed for use with the specific requirements of embedded systems in mind. Security systems for Linux or Windows are generally large, slow, and not easily ported. A product like Floodgate that is designed to be small, fast, and portable between Real-Time Operating Systems (RTOSs), on the other hand, can be easily ported between embedded systems. Floodgate has been ported to devices as small as 8-bit MCUs and can be configured to as little as 15 KB of RAM and 15 KB of ROM.

Performance is another reason to use security software designed for embedded systems. These solutions will be faster and use fewer memory resources than desktop solutions.

ECD: If embedded devices are to be deployed on a closed network, should designers consider security?

GRAU: Security needs to be designed into all embedded devices, regardless of how they will be deployed initially. Many devices originally designed for use on closed systems are later repurposed, and subsequently may be deployed on open networks. For example, many legacy SCADA systems were designed without security because they were built solely for use on closed networks. Today, many of these devices are connected to the Internet and have few, if any, security features to protect them from hackers. The result is scary; embedded devices are serving critical functions in our infrastructure and remain easy targets for hackers.

Stuxnet showed us that closed networks can still be compromised. Hackers can penetrate the network, or, as with Stuxnet, viruses, worms, and other attacks can be introduced through USB drives and other physical media. In addition, there is always the risk of insider attacks. Someone with authorized access to the network could launch an attack against devices on the network.

Enterprise networks are designed using multiple layers of security. Network firewalls protect against attacks from the Internet, security protocols protect communication, and endpoint firewalls and antivirus/antimalware software protect individual nodes on the network. Embedded devices need to follow a similar approach, adding a firewall to the device to provide an extra layer of protection, regardless of how the device will be deployed at the outset.

ECD: Are the built-in security provisions in OSs such as Android adequate for embedded applications?

GRAU: As we all know, Android runs on the Linux OS. However, many people are surprised to learn that in various Android distributions, some Linux security features have been stripped out to reduce memory usage. For example, support for packet filtering using iptables is not included in many Android distributions, meaning that firewall support is not included. So Android may not be as secure as many people believe it to be.

Security is about risk management. Hackers will break into a device or network for many reasons. Some are politically or financially motivated. Others just want to prove they can do it. The number and sophistication of attacks continue to rise. Any device with a network interface, even a device on a private network, is a potential target for attack. If the device has a Wi-Fi interface or is connected to the Internet, it almost certainly will be attacked. Devices with a Web interface will likely be targeted by automated Web hacking tools. Reports estimate that between 20 to 30 percent of all Web traffic is from hackers or other malicious packets.

As engineers should assume their devices will be attacked, they face a number of questions. How difficult can they make it for hackers to breach the device? What security measures can be put into place, and what are the costs and benefits of each of these? Five years ago security protocols such as SSH and SSL were considered enough to protect an embedded device from hackers. They are no longer sufficient. An embedded firewall is a simple and effective way to protect embedded devices from hackers capitalizing on the openness of the Internet.

Alan Grau is president and cofounder of Icon Labs.

Icon Labs alan.grau@icon-labs.com www.iconlabs.com

Follow: YouTube

M2M developers are embedded systems engineers who have teamed their skills with IT infrastructure and Internet technology. They face an emerging industry where a virtual private network of devices complicates the specific functions of individual devices. Choosing a network topology and using bandwidth efficiently are among the keys to success.

Machine-to-Machine (M2M) operation, or any smart device in general, is characterized as a device that is accessible via a network connection. With recent increases in wireless capability and coverage – cellular, wireless LAN, and Near Field Communication (NFC) – plus the rapid reduction of both hardware and connection costs, the number of machines that can be connected is increasing significantly. These machines make up a range of devices, from smart thermostats that a homeowner can access from any device with a Web browser, to smart gas meters that automatically report gas usage to the utility company, to consumer devices such as smart picture frames that automatically stream pictures stored on a website to the frame.

Current M2M capabilities are effectively the same as with any other computer operation; devices use the Web to send and receive data, usually from a cloud service or back-end server. Data is stored on these servers and then viewed by operators using general Web browsers. Mobile applications allow access to and transmission of data or commands from a smartphone or tablet to the smart device, with the cloud service or back-end server functioning as the control point in these operations.

Designers face additional challenges when implementing M2M capability. The complexities of reliable networking weigh on the credibility of device function. Network practices that enhance security, reliability, and efficiency are added to the device design requirements.

M2M design challenges

The challenges facing M2M designers today can be illustrated by comparing their tasks and responsibilities to those required of embedded systems engineers a few years ago. To build an embedded system, engineers would select the required sensors, actuators, keypad, and possibly a character display; interface them to I/O ports connected to a microprocessor; and get everything running with some combination of assembly and/or C program. They usually had the help of an in-circuit emulator and software that allowed breakpoints and single-step operation for debugging. If the device connected to anything outside, it was likely via RS-232 rather than a network. There was no real need to understand file system structure, user and group permissions, or the intricacies of network protocol. Those details were the domain of an IT department.

By contrast, M2M designers have to bridge the gap between digital/analog/software engineers and IT network engineers. They must be comfortable working on the command line of a shell. Because an M2M system incorporates an embedded processor running a modern Operating System (OS) with a network stack, M2M designers must acquire or build and patch a kernel, acquire or write recognizable device drivers, and set up a multitude of boot-up and configuration scripts. They also must add code to read sensors or write to actuators required by the application.

Another step in the M2M design process is to get connected and have a response ready when the device is dropped from the network or coming back online with a different IP address. This can be handled automatically by the OS in the case of Wi-Fi or Bluetooth on a LAN with Domain Name System (DNS), but will require intervention and specific command knowledge when RF or a cell modem is used for connection to the Internet. Power management cannot be ignored. The OS can put vital functions to sleep automatically or waste precious battery power if not set up properly for the application. File permissions and other security measures are not typically a concern for a local embedded system, but the M2M designer had better beware.

Before finishing a device design, M2M engineers must gain a working knowledge of and most likely set up the server and database tables to which the system is connecting. If M2M designers are responsible for presenting data, then working knowledge of at least CSS, HTML, PHP, and Java should be in their toolkit.

A desirable trait of an M2M designer is the ability to remain unperturbed by rapid changes. The consumer market for cell phone and tablet computing is driving the M2M industry, and the hardware and tools a designer may spend weeks or months getting familiar with will probably be updated or changed within a year. The good news is that with proper planning and care, all the work designers put into their connected devices will translate into newer, faster, and less expensive platforms in the future.

Software frameworks address M2M connectivity needs

M2M development tools are becoming more user-friendly, and strong individual and corporate support communities are available online. The OpenEmbedded software framework and Yocto Project support an array of hardware development platforms for Linux users and provide a means to manage kernel and file system work. Many engineers are already familiar with Microsoft’s .NET Compact Framework and its capabilities. Furthermore, several modem manufacturers embed powerful processors into their products, making compact COTS M2M systems possible.

Galixsys Networks provides a software framework for Linux or Android platforms that uses a Common Gateway Interface (CGI) protocol specifically designed to meet the needs of developers working with connected devices. The Andromeda framework (see block diagram in Figure 1) leverages the standard HTTP data stream to enable instant M2M communication capability and unique device identification. With a command and data payload structure, devices communicate in their natural binary, eliminating the need for markup languages. This results in greater security and reduced bandwidth with near real-time control over the Web.

While M2M developers can easily write their own service routines and access them from programs with simple system calls, Andromeda comes with a range of services such as receiving, sending, and deleting files between devices and servers. A page server command directs a server to make service calls. Reading or writing data directly to an SQL database enables developers and M2M devices to get online quickly.

The connected world is no longer coming; it has already been here for years. Thousands of new devices, phones, and tablets are connecting to the Internet every day. Across the globe, almost every industry imaginable has applications waiting to be developed to make them more efficient. How all these devices are managed and how effectively they use Web resources are the real challenges facing M2M developers.

Steve Jahnke is CTO of Galixsys Networks.

Richard Jahnke is director of engineering for Galixsys Networks.

Galixsys Networks 972-800-1301 jacquie.jahnke@galixsysnetworks.com www.galixsysnetworks.com

Follow: Twitter Facebook LinkedIn YouTube

The electronics industry has reached a point at which the dependencies between software and hardware have become so significant that they must be designed and debugged together. Efficient debug at the hardware/software interface requires full understanding of what is happening in the processor, as well as in the device registers, memory maps, and bus accesses that connect the processor to the peripherals, not to mention the internal state of these peripherals. This kind of debug capability has become crucial for delivering products successfully, at the right time, and at appropriate cost points.

While design issues at the hardware/software interface have been discussed for the better part of a decade, increased software content in today’s application-driven designs has given these issues – specifically the dependency of software on hardware and efficient partitioning – new urgency. In the past, software developers performed their debugging tasks in a hardware-independent, “peripherally blind” fashion using embedded software debuggers connected to prototype boards. This offered great insight into the processor, but little to no information about the surrounding peripherals and on-chip interconnect structures. In contrast, hardware developers have focused on lower-level effects within the registers and interconnection of Systems-on-Chip (SoCs), which are growing more and more complex every year.

When considering debug challenges, on-chip and in-system effects must be evaluated. On-chip debug needs to happen during the development phase to make sure the chip itself works correctly. In-system effects relate to how the chip behaves in its environment. Debugging in-system effects requires either complex modeling of the environment if the effects are to be considered during chip development, or control of the actual environment once the chip is available.

Figure 1 shows a typical ARM core-based SoC with a processor subsystem containing various processors linked by a coherent fabric connection to the rest of the chip. The SoC also contains custom application-specific components for 3D graphics, digital signal processing, dedicated application-specific hardware accelerators, low-speed peripherals, and high-speed interfaces. Debug challenges include debugging multiple cores in lockstep, making sure IP block integration works correctly, debugging protocols like the AMBA 4 AXI Coherency Extensions (ACE) protocol, and debugging the overall chip interconnect.

In contrast, Figure 2 shows this same SoC in its system context. Connections between the SoCs and the actual system peripherals are established on the PCB and are often based on standards like DigRF, MIPI, and USB. Now the debug challenges shift from the on-chip areas to how the chip behaves in its environment. For instance, are the frames generated by the graphics engine correctly displayed by the external display? Various off-chip and in-system effects need to be considered in tandem with on-chip effects, as they often drive graphics content and control.

Approaches to hardware/software integration and debug

During the development flow, design teams use several techniques that enable software debug and hardware/software integration.

Once all the chips are available and integrated, the hardware team will typically build a limited number of prototyping boards so the software developers can begin bringing up their code on the device. After a product is released and it proliferates, these prototype boards are often referred to as development kits. They run at real-time speed and are fully accurate. Debuggers are connected to these boards via a JTAG (boundary scan) interface. That type of software debug is very common and well understood but has its challenges, as access to the depths of hardware is limited by the level of implemented on-chip instrumentation.

FPGA-based prototypes of the chips that are to be integrated onto the board can be available several months prior to silicon. These prototypes run in the tens of MHz range, are hardware-accurate, and are often only feasible to use after stable Register Transfer Language (RTL) code is available. They allow limited debug capabilities. Connections to software debuggers are normally established via JTAG, but designers can enhance the RTL with debug information to enable hardware/software debug and analysis. Connecting the chip to the environment is possible depending on the prototype; speed rate adapters often need to be used, or the speed of the environment needs to be reduced to match the prototype speed.

Hardware emulators are available even earlier in the design flow, and they execute the chip under development – or subsets of it – in the MHz speed range. They offer fast bring-up (compared to FPGA-based prototyping, which requires more modification of the code implementing the hardware) and much better hardware/software debug because a significant portion of the hardware emulator is dedicated to debug and control of the design. However, the size and price point of today’s emulators limit their ability to be replicated to large numbers of software developers.

RTL simulation is the first execution environment in which accurate hardware and software can meet. It offers excellent hardware debug capabilities, but because it runs in the KHz range, its applicability for software development and hardware/software integration is very limited. RTL is focused on hardware verification and has traditionally been used only for very low-level, bare-metal software development. Given the complexity of modern on- and off-chip interfaces, commercial verification IP (which provides predefined test patterns to check interface correctness) can be used on-chip and within the system.

Using less accurate, abstracted hardware models, virtual chip platforms under development can run at-speed and are sometimes available 9-12 months prior to silicon. They offer excellent software debug capabilities using standard interfaces like GNU Debugger (GDB) and Cycle-Accurate Debug Interface (CADI) to connect software debuggers to virtualized hardware. The same software debuggers can be used later at the board level. Depending on modeling efforts, the full chip and its environment can be made available for advanced hardware/software debug both on-chip and within the system.

Finally, Software Development Kits (SDKs) are often the earliest available development platform. SDKs like the Apple iPhone SDK or the Android SDK enable many software developers to write their code for hardware that is very abstracted and, as such, cannot be debugged. Code developed on an SDK often needs to be recompiled to run on an actual device, in contrast to virtual prototypes and the other engines mentioned earlier, which load .elf files and run the same binary code that is later executed on the hardware target.

Debugging across the landscape of execution engines

Electronics manufacturers are increasingly distributing software across multiple cores to keep within power envelopes for complex designs. As a result, multicore debug has become a bigger challenge. Fully synchronized heterogeneous software debug for multicore designs is ideal for setting break points in all software components and the hardware itself, which then allows inspection of states, the stack, variables in the software, and registers in the hardware.

Using prototype boards, this is difficult if not impossible. If a break point triggers for the software of one processor and causes it to stop, all other processors continue to execute, changing the state of the environment in which the break point happened. In contrast, with a virtual prototype, all the participating elements – that is, all processors and hardware modules – can be stopped exactly when the break point occurred, thus allowing for efficient hardware/software debug.

In addition, when developers work on actual hardware or with older generations of virtual prototypes, they see a variety of unsynchronized debugger windows. Modern virtual prototypes allow users to efficiently integrate processor models from different vendors via abstraction layers that enable fully synchronized debug and analysis in a single, uniform environment.

Another effect that is difficult to analyze on the actual development board occurs when software has to be stopped based on conditions created by the state hardware is in. In the world of emulators, RTL simulators, and virtual prototypes, hardware debug is advanced, and both hardware and software can be efficiently halted based on break points representing the state or state transition within the hardware – like a specific counter value being reached or a specific transaction sent over a bus.

Whenever software-based hardware execution is involved, software debug can also be efficiently synchronized with a mix of different hardware abstraction levels. This is valuable at the beginning of a derivative project, for which new hardware components are available as highly abstracted models at the transaction level and not yet as hardware implemented at the RTL.

Gaining full view of hardware/software

The complexity of modern software and its dependency on the hardware upon which it is executing have made it unfeasible to delay debugging and hardware/software integration until all the chips are available and integrated onto the PCB. Several execution engines are available to chip and system development teams, but the capability to develop and debug software varies greatly among these engines. Figure 3 shows the previously introduced chip and board combined with engines to execute the chip under development as well as connections to hardware/software debug.

Debug has several layers, often built on Integrated Development Environments (IDEs) like Eclipse. Users need to debug the actual hardware, the bare-metal software execution outside of operating systems, and the hardware and software in combination, as well as the performance of the overall system.

With the hybrid combination of different engines and a new generation of software debuggers, the industry is approaching an era in which software developers can get a complete programmer’s view of the software and hardware much earlier in the design cycle than they ever could before.

Frank Schirrmeister is senior director of product marketing for the System and Software Realization Group at Cadence Design Systems.

Michael (Mac) McNamara is VP and general manager of system-level design at Cadence Design Systems.

Larry Melling is the product marketing manager for the Cadence Virtual System Platform at Cadence Design Systems.

Neeti Bhatnagar is the engineering director for the Cadence Virtual System Platform at Cadence Design Systems.

Cadence Design Systems 408-348-7025 franks@cadence.com mcnamara@cadence.com lmelling@cadence.com neeti@cadence.com www.cadence.com

Follow: Twitter Facebook LinkedIn YouTube

Although capable of running embedded software through emulation and simulation, traditional electronic design automation tools do not focus on the critical concerns of embedded systems design. This deficiency, in addition to software developers’ unfamiliarity with hardware tools, requires a new approach. Software developers can interact with hardware earlier by implementing the same native tools used to develop applications that run on FPGA prototypes and final silicon. This allows for early investigation, which improves hardware design and accelerates software debug and bring-up for embedded systems.

Long gone are the days when embedded software development commenced when hardware was ready. The ever-increasing size and complexity of software necessitate an earlier starting date to have a chance of shipping on time. Companies must ensure proper software integration ahead of silicon tape-out given the investment and risk associated with software development. Ever-shortening time-to-market windows further exacerbate these challenges.

The only solution is to look for ways to execute, and hence, test and debug software before the final target hardware is available. Development often must start in earnest even before the hardware design is finalized. One way to improve hardware design and verification and accelerate software bring-up is to deploy the application on hardware that is either virtual or emulated.

Virtual prototypes

There is no single, precise definition of the term “virtual prototype.” For the purposes of this article, the term will be used to describe any environment in which embedded code may run, thus enabling useful development prior to the availability of actual target systems. Let’s take a closer look at each of these possibilities.

Native code running on a PC

It’s an obvious first step to compile and run code on a PC. Tools are readily available, cheap, or even free, and a PC offers high levels of functionality. This environment is fine for testing algorithms and basic logic. The code will probably run faster on a PC compared to the real target, as a PC’s CPU is likely to be more powerful than an embedded processor.

Apart from clock speed, code timing is not useful, as the instruction mix on an x86 processor is very different from most embedded devices. As soon as the code needs to interact with hardware or the Real-Time Operating System (RTOS), this execution environment stops being useful.

Native code run with peripheral/system models

Most RTOS manufacturers provide a host execution environment that enables either a special version of the RTOS to be run or its functionality to be emulated under Windows. This, along with a means of providing functional models of peripherals, enables further progress to be made. Timing is still misleading, however.

Execution on an evaluation board

Most semiconductor vendors provide low-cost evaluation boards to facilitate fast deployment of their CPUs. Commercial RTOS products such as Mentor Embedded’s Nucleus RTOS may be available preconfigured for such boards, thus enabling rapid production.

This execution environment is attractive, as the CPU speed and instruction mix is likely to be very close to the final target, which makes the testing of time-sensitive code viable. The accuracy of such an environment depends upon the similarity of the peripheral devices to those on the target.

Use of an Instruction Set Simulator

Although executing a real chip’s code seems attractive, it has the drawback of requiring code to be added in order to gain visibility of some software functionality. This is called “instrumenting” the code. An alternative approach is to use an Instruction Set Simulator (ISS), which simulates code execution on an instruction-by-instruction basis. An ISS can run close to real-time speed and offers precise, highly visible code execution. In effect, real time can be stopped as the ISS tracks clock cycles consumed during simulation.

Most ISS products allow some kind of functional peripheral modeling, which allows significant progress to be made with software development.

An ISS with Hardware Description Language models of peripherals (co-simulation)

Hardware is designed using a Hardware Description Language (HDL) such as VHDL or Verilog. Designers routinely use simulators to verify their HDL designs, and many of today’s development tools merge an ISS with an HDL simulator. This enables code to be executed in an accurate CPU environment that interacts with what appears to be real hardware. The software developer can use the HDL models of the final target system to develop software components such as drivers and boot code that interact closely with the hardware.

The downside of co-simulation is that greater precision comes at the cost of reduced execution speed.

An HDL model of the complete system, including CPU

It would seem logical that the most accurate virtual prototype would be an HDL model of the complete system, including the CPU and peripherals. Three reasons explain why this is not really the case:

1. The code execution speed on such a model would be glacial. It would not be fast enough to get anything useful done.
2. An HDL for the CPU is unlikely to be available.
3. Since an ISS is likely to be designed carefully, its use does not have any downside, but does increase performance to a useful level.

An ISS with SystemC models of peripherals

To allow proper simulation speed that can accommodate software execution, a system can be modeled with higher abstraction languages such as SystemC (C/C++ class library). Modeling at higher abstraction levels uses loose or approximated timing. Such timing is appropriate for software execution and performance analysis.

Hardware emulation

The virtual prototyping technologies discussed thus far can be plotted on a graph of code execution speed against precision and essentially yield a straight line (see Figure 1). Developers can choose from a range of possibilities: fast, abstract simulation at one extreme and slow, exact simulation at the other. However, another technology bucks this trend and strays away from the straight line.

Although the speed limitations of HDL simulation can be reduced by simply using a powerful desktop computer, this has limits and designers always want more. The response from the Electronic Design Automation (EDA) community was to develop emulation. An emulator is specialized hardware that, in effect, offers a dedicated environment to run an HDL simulation. This is typically achieved using FPGAs.

An integrated platform built with an ISS, SystemC models, and an emulator that simulates some of the peripheral hardware breaks the mold and provides a precise, high-performance execution environment. Running a virtual target and emulation offers much deeper visibility into hardware and software execution threads and enables more efficient debug as well as system performance analysis.

Beyond debug

Embedded software developers traditionally have focused on getting their code to function correctly. At the highest level of abstraction, this results in the device responding to stimuli in a predicable fashion in line with design specifications. This has not changed, but the developer’s brief is becoming wider. The most significant addition to the software developer’s workload is the consideration of power.

Low-power design is topical for several reasons. While this historically has been a hardware issue, today’s complex designs offer numerous opportunities for power consumption to be tuned according to the system’s current state, software, and real-time context. That state is determined by the software; hence, power management is becoming a software issue.

It is a tall order to develop and debug power management code ahead of hardware availability using a virtual prototype, but that is exactly what is required. Of course, it is all possible in principle; hardware simulation can yield power consumption figures, and the actual power consumed by a CPU can be measured. It is simply a question of communicating this information to the software developer in a meaningful way.

The way forward

Any thought of software and hardware development being separate activities must be dismissed. The good news is that System-on-Chip (SoC) producers now recognize the need for embedded software development ahead of silicon. The bad news is that although traditional EDA hardware tools can run embedded software through emulation and simulation, they do not focus on the critical concerns of embedded systems design, including operating system context, multicore and thread handling, and caching considerations.

An integrated approach is needed to provide tools that are engineered to work in a well-coordinated fashion and present information in a way that is familiar to both hardware and software teams. Software developers must be able to interact with hardware earlier using the same native tools they leverage to develop applications that will run on FPGA prototypes and final silicon with integrated technologies. One such unified approach is available in the Mentor Embedded Platform, which incorporates familiar technologies from Mentor Graphics such as virtual prototyping using Vista hardware debug and analysis and the Sourcery CodeBench Integrated Development Environment (IDE) for software development. By using this integrated embedded platform for early software development, developers can conduct performance analysis with virtual and emulated hardware, as well as investigate cache, process, thread, and core activities before silicon is available.

This early investigation between disciplines improves design hardware and accelerates software debug and bring-up for SoCs and embedded systems. Software developers and hardware engineers can all agree this is a move in the right direction.

Colin Walls is a member of the marketing department at Mentor Graphics Embedded Software Division.

Mentor Graphics Embedded Software Division colin_walls@mentor.com www.mentor.com

Follow: Twitter Blog LinkedIn

Modern ASICs and FPGAs are tedious and time-consuming to verify and validate. Adding small, highly efficient on-chip capture infrastructure to the design makes this job much easier by providing 10x the visibility of instrumentation points for a given area. In addition, by using compression algorithms, logic analysis capture stations can capture data for 10x or more capture depth.

ASICs and FPGAs have become massively complex, particularly for System-on-Chip (SoC) designs involving multiple cores. With this complexity comes longer and more tedious debug and validation cycles. Unfortunately, when something fails or goes wrong, gaining access to test points in highly integrated designs is next to impossible. Unless you want to spend weeks shooting in the dark at random errors while running through multiple prototypes, on-chip instrumentation is no longer optional; it’s a critical must-have. Figure 1 shows an overview of the debug process using on-chip instrumentation.

While there are a number of ways to add instrumentation to FPGAs, a distributed approach using an instrument network is emerging as the preferred method, as it maximizes the number of potential observation points while minimizing silicon area or look-up table utilization requirements. Also critical to efficient debug is deep trace capture to see how the various parts of a system interact over time. Finally, designers must be able to observe the interactions of multiple devices and clock domains, both on- and off-chip, all fully time correlated for a true system-level perspective.

Taken together, innovations including flexible and complete access to observation points, deep trace captures, and system-level views have the potential to change the game for FPGA and ASIC debug from long and arduous to fast and efficient.

Debug challenges

Before reviewing different approaches to implementing embedded instrumentation, it’s helpful to understand why instrumentation is necessary in the first place. The biggest reason is simply the ever-growing functionality in each system.

Whereas in the past there were plenty of probe points (external I/O on the devices) to choose from, it’s no longer possible to observe what’s going on since in most cases the key interfaces are now inside devices. Current-generation FPGAs have 100x the number of functions running in parallel compared to five years ago, yet the number of external outputs has stayed the same. From the perspective of a developer attempting to debug unexpected behavior, modern chips are nothing more than a big black box.

As if that weren’t enough, while the power of simulators continues to improve in a linear fashion, every increase in parallel functionality adds an exponential increase in potential combinations. Since simulations run on one combination at a time, it’s not possible to cover all the functionality in pre-silicon simulation runs.

This inability to adequately simulate all the possible permutations in pre-silicon has led to FPGA-based prototyping before design completion. Particularly at the prototype level, access to observation points is extremely helpful in debugging functional issues quickly and efficiently.

Another debug challenge is the emergence of embedded software on silicon. More and more FPGA and ASIC designs include one or more processor cores. Such systems can include a complex mix of software, firmware, embedded processors, GPUs, memory controllers, and other high-speed peripherals. This increased functional integration combined with faster internal clock speeds and complex, high-speed I/O is making it harder than ever for developers to deliver a functional and fully validated system.

On-chip signal capture

Back when systems involved multiple chips and components it was easy to move logic analyzer probes around to look at different signal combinations. Even with the move to on-chip instruments, the need to flexibly move virtual logic analyzer probes to different signal points remains a constant. Since the designer can’t anticipate every variable or potential application for a given chip, the more signal capture points available, the better.

A traditional ASIC approach uses a mux network with shared select signals (one per mux level) and provides n/m different signal combinations, where n is the number of probe points and m is the number of signals viewed concurrently (debug bus width). This is the most restrictive but simplest option, as it leverages simple multiplexers. To be effective, this approach requires significant up-front time to create groups of signals that correspond to every possible debug scenario, and once the capture points are in play, designers can only look at signals that are in the same group. This process is demanding, time-consuming, and highly unlikely to capture all debug scenarios.

The other extreme is to create a full crossbar mux that gives complete signal flexibility, which requires m muxes of n:1 in size. This can get expensive relative to area very fast, making this approach impractical for all but the smallest cases.

The middle ground is to either increase the number of select signals inside the mux structure or create a number of duplicate groups with different signal ordering. The shared select mux and the mux with additional select signals are both implemented in many homegrown approaches. While shared select mux schemes can handle common and expected debug scenarios, they still fall short of the ideal complete coverage. Thus, they are ill-suited for unexpected problems and can often lead to inefficient implementations, as signals are repeatedly connected to multiple multiplexers.

It is possible to find a more elegant and efficient solution by leveraging multistage, unordered networks, often called concentrator networks. This new approach effectively creates an observation network and is becoming commercially available. Using a unique network architecture and complementary routing algorithms, an observation network provides the signal flexibility of a full crossbar mux while in most cases requiring no more die area than shared simple muxes. Table 1 shows a comparison of signal visibility calculated using the different approaches.

With an observation network, designers use automated tools to implement on-chip signal capture probes in the Register Transfer Language (RTL). At the design stage, there is no need to worry about different signal combinations or ordering since every combination will be available. The result is an observation network that grows linearly with the number of signals. This approach moves the complexity of determining routing off the silicon and into software. While producing a significant area/performance improvement, an observation network requires sophisticated algorithms to determine routing, making it difficult to use without commercial software to control signal selection.

How significant an advantage does the network approach offer over a simple mux in terms of observation point visibility? Take this example, where 256 signals are probed (n) with 32 signals visible concurrently (m):

• Simple mux: Number of signal combinations (visibility) = 256/32 = 32
• Observation network: Number of signal combinations (visibility) = 2^256 = 1.2 x 10^77

The difference is 76 orders of magnitude. While the first approach is highly restrictive, the observation network approach provides any possible combination of signals. For roughly the same cost, the observation network provides a huge advantage with its increased flexibility.

Maximizing capture depth

For debug challenges that span hardware and software, the ability to capture long traces is critical to track down problems that show up over thousands or millions of clock cycles. Post-silicon and on FPGAs, deep capture is vital to see how the overall system works, as many of the bugs that escape verification take a long time to emerge. Furthermore, most software-driven functionality spans hundreds of thousands to millions of clock cycles.

Traditional instrumentation approaches capture the information as it is received from the observation probe using one entry in the internal RAM for each clock cycle of data captured. With this approach it is difficult or impossible to capture more than a few thousand clock cycles at a time without putting an unacceptable strain on internal memory resources. For that reason, compression techniques are now starting to be used to boost capture depth.

However, most well-known compression algorithms are poorly suited to trace compression, having been developed for visual media and communications applications. Specialized trace compression layers that use multiple compression techniques together, each specifically tailored to common trace data patterns, are now commercially available. For most real-life applications this provides 10-1,000x more depth with no loss in resolution.

Efficient system-wide debug

The last piece of the puzzle to more efficient FPGA and ASIC debug is a time-correlated, system-wide view that spans multiple clock domains running in parallel. When problems require correlation across multiple instrumented areas, the designer is looking at a time-consuming process of obtaining individual traces and then correlating events manually. For instance, an average ASIC prototype on an FPGA-based prototyping platform consists of two to three clock domains per FPGA across four to eight FPGAs. This means the designer will need to debug anywhere from eight to 24 clock domains individually. Tracing each of these 24 domains one at a time and manually piecing together the results is time-consuming and error-prone.

A much more efficient approach is to use logic analyzer software to produce a time-correlated view from independent instruments operating in multiple clock domains and across multiple devices, as shown in Figure 2. Specialized debug software can collect data from each instrumented area of the chip, reverse the compression algorithms, and then align the captured data to produce a system-wide, time-correlated view. This leads to a single trace capture and debug scenario, both saving time and providing simultaneous hardware debug of many functional units and clock domains. This process often reveals emergent system behaviors that were never considered when the device was architected.

Innovations handle the unexpected

With increasing complexity and limited access to probe points, ASIC and FPGA validation and debug has become tedious and time-consuming. As more and more functionality is integrated into each chip, physical access to probe points has become impossible. The challenge then is to incorporate enough on-chip observation points to not only handle expected debug scenarios, but unexpected ones as well.

A key innovation that enables faster and more efficient validation and debug of even the most complex designs is an observation network. Compared to traditional shared select mux approaches for observing signals, the observation network delivers significantly more signal combinations with similar die area requirements.

Other innovations supporting more efficient debug scenarios include the use of advanced compression algorithms to boost on-chip memory capture depth and the emergence of logic analyzer software that produces a time-correlated, system-wide view that spans multiple devices and off-chip instruments.

Brad Quinton is the chief architect for the Tektronix Embedded Instrumentation Group.

Tektronix brad.quinton@tektronix.com

www.tektronix.com

Follow: Twitter Facebook Google+ LinkedIn YouTube

Although capable of running embedded software through emulation and simulation, traditional electronic design automation tools do not focus on the critical concerns of embedded systems design. This deficiency, in addition to software developers’ unfamiliarity with hardware tools, requires a new approach. Software developers can interact with hardware earlier by implementing the same native tools used to develop applications that run on FPGA prototypes and final silicon. This allows for early investigation, which improves hardware design and accelerates software debug and bring-up for embedded systems.

Long gone are the days when embedded software development commenced when hardware was ready. The ever-increasing size and complexity of software necessitate an earlier starting date to have a chance of shipping on time. Companies must ensure proper software integration ahead of silicon tape-out given the investment and risk associated with software development. Ever-shortening time-to-market windows further exacerbate these challenges.

The only solution is to look for ways to execute, and hence, test and debug software before the final target hardware is available. Development often must start in earnest even before the hardware design is finalized. One way to improve hardware design and verification and accelerate software bring-up is to deploy the application on hardware that is either virtual or emulated.

Virtual prototypes

There is no single, precise definition of the term “virtual prototype.” For the purposes of this article, the term will be used to describe any environment in which embedded code may run, thus enabling useful development prior to the availability of actual target systems. Let’s take a closer look at each of these possibilities.

Native code running on a PC

It’s an obvious first step to compile and run code on a PC. Tools are readily available, cheap, or even free, and a PC offers high levels of functionality. This environment is fine for testing algorithms and basic logic. The code will probably run faster on a PC compared to the real target, as a PC’s CPU is likely to be more powerful than an embedded processor.

Apart from clock speed, code timing is not useful, as the instruction mix on an x86 processor is very different from most embedded devices. As soon as the code needs to interact with hardware or the Real-Time Operating System (RTOS), this execution environment stops being useful.

Native code run with peripheral/system models

Most RTOS manufacturers provide a host execution environment that enables either a special version of the RTOS to be run or its functionality to be emulated under Windows. This, along with a means of providing functional models of peripherals, enables further progress to be made. Timing is still misleading, however.

Execution on an evaluation board

Most semiconductor vendors provide low-cost evaluation boards to facilitate fast deployment of their CPUs. Commercial RTOS products such as Mentor Embedded’s Nucleus RTOS may be available preconfigured for such boards, thus enabling rapid production.

This execution environment is attractive, as the CPU speed and instruction mix is likely to be very close to the final target, which makes the testing of time-sensitive code viable. The accuracy of such an environment depends upon the similarity of the peripheral devices to those on the target.

Use of an Instruction Set Simulator

Although executing a real chip’s code seems attractive, it has the drawback of requiring code to be added in order to gain visibility of some software functionality. This is called “instrumenting” the code. An alternative approach is to use an Instruction Set Simulator (ISS), which simulates code execution on an instruction-by-instruction basis. An ISS can run close to real-time speed and offers precise, highly visible code execution. In effect, real time can be stopped as the ISS tracks clock cycles consumed during simulation.

Most ISS products allow some kind of functional peripheral modeling, which allows significant progress to be made with software development.

An ISS with Hardware Description Language models of peripherals (co-simulation)

Hardware is designed using a Hardware Description Language (HDL) such as VHDL or Verilog. Designers routinely use simulators to verify their HDL designs, and many of today’s development tools merge an ISS with an HDL simulator. This enables code to be executed in an accurate CPU environment that interacts with what appears to be real hardware. The software developer can use the HDL models of the final target system to develop software components such as drivers and boot code that interact closely with the hardware.

The downside of co-simulation is that greater precision comes at the cost of reduced execution speed.

An HDL model of the complete system, including CPU

It would seem logical that the most accurate virtual prototype would be an HDL model of the complete system, including the CPU and peripherals. Three reasons explain why this is not really the case:

1. The code execution speed on such a model would be glacial. It would not be fast enough to get anything useful done.
2. An HDL for the CPU is unlikely to be available.
3. Since an ISS is likely to be designed carefully, its use does not have any downside, but does increase performance to a useful level.

An ISS with SystemC models of peripherals

To allow proper simulation speed that can accommodate software execution, a system can be modeled with higher abstraction languages such as SystemC (C/C++ class library). Modeling at higher abstraction levels uses loose or approximated timing. Such timing is appropriate for software execution and performance analysis.

Hardware emulation

The virtual prototyping technologies discussed thus far can be plotted on a graph of code execution speed against precision and essentially yield a straight line (see Figure 1). Developers can choose from a range of possibilities: fast, abstract simulation at one extreme and slow, exact simulation at the other. However, another technology bucks this trend and strays away from the straight line.

Although the speed limitations of HDL simulation can be reduced by simply using a powerful desktop computer, this has limits and designers always want more. The response from the Electronic Design Automation (EDA) community was to develop emulation. An emulator is specialized hardware that, in effect, offers a dedicated environment to run an HDL simulation. This is typically achieved using FPGAs.

An integrated platform built with an ISS, SystemC models, and an emulator that simulates some of the peripheral hardware breaks the mold and provides a precise, high-performance execution environment. Running a virtual target and emulation offers much deeper visibility into hardware and software execution threads and enables more efficient debug as well as system performance analysis.

Beyond debug

Embedded software developers traditionally have focused on getting their code to function correctly. At the highest level of abstraction, this results in the device responding to stimuli in a predicable fashion in line with design specifications. This has not changed, but the developer’s brief is becoming wider. The most significant addition to the software developer’s workload is the consideration of power.

Low-power design is topical for several reasons. While this historically has been a hardware issue, today’s complex designs offer numerous opportunities for power consumption to be tuned according to the system’s current state, software, and real-time context. That state is determined by the software; hence, power management is becoming a software issue.

It is a tall order to develop and debug power management code ahead of hardware availability using a virtual prototype, but that is exactly what is required. Of course, it is all possible in principle; hardware simulation can yield power consumption figures, and the actual power consumed by a CPU can be measured. It is simply a question of communicating this information to the software developer in a meaningful way.

The way forward

Any thought of software and hardware development being separate activities must be dismissed. The good news is that System-on-Chip (SoC) producers now recognize the need for embedded software development ahead of silicon. The bad news is that although traditional EDA hardware tools can run embedded software through emulation and simulation, they do not focus on the critical concerns of embedded systems design, including operating system context, multicore and thread handling, and caching considerations.

An integrated approach is needed to provide tools that are engineered to work in a well-coordinated fashion and present information in a way that is familiar to both hardware and software teams. Software developers must be able to interact with hardware earlier using the same native tools they leverage to develop applications that will run on FPGA prototypes and final silicon with integrated technologies. One such unified approach is available in the Mentor Embedded Platform, which incorporates familiar technologies from Mentor Graphics such as virtual prototyping using Vista hardware debug and analysis and the Sourcery CodeBench Integrated Development Environment (IDE) for software development. By using this integrated embedded platform for early software development, developers can conduct performance analysis with virtual and emulated hardware, as well as investigate cache, process, thread, and core activities before silicon is available.

This early investigation between disciplines improves design hardware and accelerates software debug and bring-up for SoCs and embedded systems. Software developers and hardware engineers can all agree this is a move in the right direction.

Colin Walls is a member of the marketing department at Mentor Graphics Embedded Software Division.

Mentor Graphics Embedded Software Division colin_walls@mentor.com www.mentor.com

Follow: Twitter Blog LinkedIn

Customization and adaptability are important considerations when designing industrial and consumer electronics. Although ASICs and reference designs can be a good starting point for a system, lack of flexibility makes it hard to use ASICs for a complete solution. Using software for customization, even when it involves hardware interfaces, enables unrivalled design flexibility and thus greater product differentiation.

To understand why customization is important, consider the following three examples where a reference design can be used as a starting point, but the designer might want to make changes to the design, even at a late stage:

1. A design implements part of a communications stack (for example, USB audio), but the hardware interface on the audio side can be customized to communicate with a specific type of codec such as I2S master or slave S/PDIF, or it can be customized with an extra control endpoint.
2. An industrial controller implements one of the many standard protocols; for example, a vendor of industrial motors might want to include an Ethernet PHY, but defers the decision whether to run EtherCAT or any other real-time Ethernet stack.
3. An embedded Web server can be used to configure either of the previous examples, provided that the designer can implement the control logic inside the Web server.

The ease with which designs can be customized depends on the technology used to implement the core of the design. Did it use an ASIC, an FPGA, or a real-time processor?

If the design is implemented as an ASIC, then the only feasible way to customize it is to modify parameters built into the ASIC. Considering the first example, a USB audio ASIC will have a method to set the product name of the USB device and might provide a choice between left- and right-aligned I2S. However, it might not offer the options to add an extra control endpoint or to use S/PDIF as an output interface. Indeed, it is not possible to develop an ASIC that supports all plausible interfaces, and it is not economical to develop a family of ASICs that supports well-chosen, small sets of interfaces.

If the design is implemented in an FPGA then, in theory, customization can be as extreme as the user wants it to be. However, a hardware design flow might make customization a not-so-straightforward process. Implementing an additional interface such as S/PDIF is not difficult for hardware designers, but implementing an extra control endpoint requires software to be re-implemented as a piece of hardware. Furthermore, the result must be synthesized, and until the customized design is completely synthesized, placed, and routed it is not clear as to whether the original timing constraints are all still met.

The third option is to implement the design entirely as software. This is often considered difficult to accomplish because of the many conflicting real-time requirements involved. However, a software implementation can be perfectly manageable, provided that the programmer can split the problem into a set of independent real-time tasks that run on a bucket of independent real-time processors.

This gives the designer the flexibility to implement a hardware protocol simply as a software task by dedicating one of the real-time processors to implementing that hardware protocol. For example, one can implement I2S in software by wiggling the clock and data lines in an appropriate manner. Also, USB can be implemented by reading and writing data to the USB PHY. If these two activities are executed as two independent tasks, then all real-time deadlines can be met individually.

The latter is a programming model known as concurrent real-time programming. Figure 1 shows how a single XMOS XCore processor can execute eight threads simultaneously. Each thread can be seen as an individual processor with a guaranteed real-time execution rate. Each instruction will execute within a known amount of time; hence the programmer can predict whether programs will meet timing deadlines or not.

The beauty of this model is that timing prediction is entirely self-contained to the task. As an example, consider the I2S task mentioned before. Once the program has been written in a way that it will execute on, for example, a 50 MIPS processor, then regardless of modifications to other threads, this I2S thread will always meet its timing. The only way to break the timing is by not providing data at the appropriate rate. This is to be expected; if data is not provided at 48 kHz, then the codec cannot be given the data at 48 kHz, and something must break.

Software is not a panacea. Two obvious limitations are interfaces that require too much instantaneous bandwidth and hardware interfaces that need very short turnaround times. For example, it would not be feasible to write a multi-GHz SERDES in software on today’s processing hardware, but many commonly found interfaces can be formulated in software.

There are three reasons why a software implementation is preferable over a hardware interface:

1. Economics: It is not possible to provide all interfaces available as hardware blocks. A processor can model any type of interface as required.

2. Adaptability: An interface built in hardware is set in silicon. If a standard evolves, or if a particular device is not completely compliant or is required to perform some extra operations, then a silicon interface will not do the job. A software-defined interface can be adapted to meet the requirements.

3. Optimization: A software-defined interface can be optimized to address the problem at hand. Consider the EtherCAT application mentioned previously. If a hardware Media-Independent Interface (MII) is used, then the packet is stored and forwarded, which does not meet, for example, latency requirements. If the interface is defined in software, then the buffering requirements can be tailored to the problem.

The third reason is important, as many standard blocks dealing with UARTs, I2S, MII, and other standards must have built-in FIFO buffers to decouple the interface, and those buffers in many cases are unwanted because they add delay.

Once the designer chooses a software-based design flow, the possibilities are endless. Figure 2 shows a USB board that simultaneously supports an analog interface using a stereo I/O codec (over I2S), a differential digital interface (using S/PDIF), and a Musical Instrument Digital Interface (MIDI).

The embedded processor implements the USB stack and the audio-over-USB protocol. In this particular configuration, three audio interfaces are driven: MIDI, I2S/analog, and S/PDIF. The latter can be changed to a different protocol such as ADAT by a mere software change. More importantly, variations of the product can be made with different sets of interfaces, such as a multitude of coaxial outputs or I2S-based codecs. The software on the embedded processor can be tailored to the specifics of the interfaces with only minor alterations. There are no predefined numbers of each interface.

Flexible hardware can enable greater product differentiation, allowing developers and product managers to deliver distinguishable value to their customers. Concurrent real-time programming allows tight timing requirements to be guaranteed, with embedded processors such as the XMOS XCore providing the silicon and development tools required to take advantage of this programming method.

Henk Muller is principal technologist at XMOS Ltd.

XMOS Ltd. henk@xmos.com www.xmos.com

Follow: Twitter Facebook Google+ LinkedIn YouTube Community forum

Peer review can help embedded hardware designers and software engineers find and fix problems earlier in the development cycle, saving money and time. As opposed to a manual approach, tool-assisted peer review provides reportable data that enables development teams to benchmark and improve their processes.

It’s a common scenario in the software development world: A company pins a lot of hope on a new device, hoping to ride the wave of progress in the industry by getting it to market ahead of the competition. The hardware and software teams work long and hard to make it happen. Once the product reaches consumers, though, a problem quickly arises: it doesn’t integrate well with a market-leading peripheral.

As engineers think back to where things might have broken down, they realize it couldn’t have been during the spec review phase because they routinely review requirements documents, specifications, and test cases. Therefore, the problem most likely occurred during the coding process (see Figure 1). And it probably happened because the software and hardware teams were out of sync and unaware of changes during the design and development phase because they didn’t have a simple way to stay connected.

Hardware designers and software engineers who have been working with embedded code for any length of time have probably encountered a situation like this. If not, it’s certainly an embedded developer’s worst fear: A problem with a product that has already reached the marketplace or is about to be released. It could be an integration issue, a connectivity problem, or a security glitch. Whatever the problem, the outcome is the same; time and money are lost while the code is fixed and the company’s reputation suffers.

When so many embedded teams have experienced these problems, why are they still occurring? The reasons are simple: lack of peer and code review and lack of collaboration between software and hardware teams.

Although reviewing artifacts early in the design phase is a common practice and known to be the best way to detect problems early, automating this process between hardware and software engineers has largely been impossible. Although few would disagree that code reviews are always a good idea, other pressures often take precedence. Of course, there are exceptions, mainly in regulated and safety-critical products. However, according to embedded code expert Jack Ganssle, about 98 percent of embedded developers aren’t doing peer review. That’s a stunning statistic given what’s at stake.

While ignoring code review might have worked in a less complicated, less connected world, using that method today leaves companies open to security, interoperability, and connectivity issues. That can lead to expensive recalls requiring time-consuming fixes after products have reached the market.

Peer review – a process by which team members inspect design documents, artifacts, and source code – helps both software and firmware developers, as well as hardware designers, find more bugs or related design errors earlier in the design and prototyping stages, improving product quality and minimizing costly rework later in the development process. It’s a process where software and firmware developers as well as the hardware design teams can share and review both technical documents and programming code in a timely manner, keeping the teams in sync when issues are found and changes are made.

Put simply, studies show that peer reviews work. Research by Philip Koopman, an Associate Professor from Carnegie Mellon University, found that peer reviews are the most cost-effective way to find bugs, and 40 to 60 percent of defects are found by such reviews. Koopman also found that reviews cost only about 5 to 10 percent of the project cost.

Peer review helps ensure:

• Higher-quality products in the short term as defects are identified
• Higher-quality products in the long term as technical debt is better managed
• Compliance with applicable regulations
• Interoperability with all potential products, peripherals, and software it may be used with
• Crisper, better documented, and better organized code
• Transfer of knowledge across the entire development team

The peer review process also saves money, as evidenced by a study comparing bug defects before and after code review authored by SmartBear Software in conjunction with Cisco Systems. In both cases, the product had 463 bugs remaining after development. Without code review, getting the bug count down to 194 cost $368,000. The code review process not only fixed more bugs, getting the bug count down to 32, but it did so for $152,000.

Tool-assisted peer review makes sense

On the surface, manual methods of peer review seem like a good way to introduce peer review without spending the money for an automated tool. But these manual methods, while certainly better than no peer review at all, are time-consuming. Furthermore, it can be difficult to collaborate with team members in different locations or time zones. What’s more, manual methods like ad hoc meetings, water cooler discussions, sending code snippets or PDFs via e-mail, and cutting and pasting code into Word documents tend to be disorganized, and critical points can be lost in the process.

Another consideration is that manual peer review does not produce reportable data. One key to creating support for process improvement is quantifiable results. This is one reason teams wonder if the hours spent in meetings are really worth it.

Automated peer review (also called tool-assisted peer review) solves these problems. With tool-assisted peer review, hardware designers and software engineers can participate in reviews at any time, not on a set schedule. That saves time and increases engineer productivity. Developers also can share and collaborate with team members in different locations. And because all materials are in one place, gathering the right files and design documents is never a problem. In addition, having the review materials and results managed in a reportable database, as well as providing accountability within the review process, helps adhere to multiple regulatory compliance mandates.

Tool-assisted peer review is more efficient and effective than manual peer review. It enables software developers and hardware engineers to catch defects, whether stand-alone or based on changes one team must make that affects the other, earlier in the development process at a time when they are easier and faster to fix. The general rule of thumb is that defects detected later in the process take longer and are more complicated and expensive to fix.

Tool-assisted peer review also provides developers with a host of standard and customizable reports on metrics like defect density, inspection rate, defect detection rate, recent and open defects, lines of code added/modified/deleted, and reviews by change list. These reports and metrics can make a significant difference in the software and hardware development process. With the right metrics, the respective teams can benchmark and improve their processes. For example, it could flag reviews considered trivial or stalled, saving the team valuable time.

Some tools, like SmartBear’s PeerReview Complete, also allow development teams to use a variety of review formats such as Word and PDF documents, 2D drawings, schematics, VHDL code and images; develop custom workflows; create custom reports and metrics; integrate with Eclipse and Visual Studio; create customizable fields for tracking and reporting key Capability Maturity Model Integration (CMMI) audit metrics; and implement administrative and security controls. It also integrates with a development team’s existing issue tracking, development environments, and version control tools. A schematic review is shown in Figure 2.

Tool-assisted peer review for all development artifacts

With these types of capabilities, tool-assisted peer review tools can serve as a comprehensive solution that works with code and all artifacts created during the development process. Requirements documents, hardware and software design documents, schematics, 2D drawings, and test specifications can all be reviewed using the same time-saving tool.

The entire review process comes together in one place, simplifying the existing document review process and extending the review process into code review. The long-established benefits of peer review for design documents can be expanded to the coding process. Code review becomes a normal part of the development cycle, and early detection of defects in code becomes as natural as early detection of defects in design specifications.

A move toward peer review is a positive one for any embedded development team. It eliminates guesswork, improves productivity, saves money, and streamlines workflows. While it generally takes time to implement even a manual review process, tool-assisted peer review provides an immediately impactful peer review process without the headaches of traditional manual approaches.

John Lockhart is the product manager of PeerReview products for SmartBear Software.

SmartBear Software john.lockhart@smartbear.com | www.smartbear.com

Follow: Twitter Blog Facebook Google+ LinkedIn YouTube

While model-based development has helped developers identify defects earlier and cope with increasing design complexity, testing is now the elephant in the room. How can automation, integration, and collaboration around testing deliver the required efficiencies? New approaches are moving the embedded testing challenge from code to models, allowing businesses to gain a competitive edge as a result.

An age of smarter products is ushering in embedded product designs with increased functionality, rising complexity, and compressed delivery windows. These products often need to comply with strict development regulations for use in safety- or mission-critical applications such as aircraft, automobiles, or medical devices.

Model-based development has boosted developer productivity with graphical notations such as Unified Modeling Language (UML) and Systems Modeling Language (SysML) to help manage complexity and uncover design anomalies earlier in the development life cycle. In many cases this has left testing struggling to catch up. It is frequently the biggest time and budget item in projects, and therefore the first to be cut. Despite this, testing costs are increasing; much testing has remained rooted in manual, code-based approaches that do not easily scale to today’s demands.

The proven power of model-driven development

Using models, software engineers can more clearly understand and analyze requirements, make architectural trade-offs, define design specifications, validate and verify behavior with simulation, and generate code for direct deployment on target hardware. A key benefit of a model is that consistency and correctness are maintained across the design as it changes. With UML, each diagram can capture different views of the model at different levels of abstraction while remaining consistent across these views. The semantics of the modeling language enforce team consistency and help automate life-cycle tasks, including code generation.

Repeatable processes are the key to improving collaboration and productivity while reducing the cost of demonstrating regulatory compliance. A 2011 embedded development study by Jerry Krasner of Embedded Market Forecasters found that model-driven development reduced development time typically lost to delays by more than 40 percent, delivering typical project savings of $250,000.

Enhancing quality at every step of the life cycle

The next natural step to maximize productivity and agility is to improve the testing process. The “usual suspects” addressing this are the late detection (and resolution) of defects, as well as the communication and consistency issues between teams as changes occur and defects are detected.

Early identification of defects is critical to reduce development costs and meet time-to-market objectives. Many defects are introduced in the design during the early stages of development, but repair costs increase exponentially as defect resolution is delayed. Not only can late defect resolution harm project financial performance, a high intensity of late rework can also derail delivery schedules and delivered quality. Defect resolution must be closely linked to defect discovery, as it is typically much easier to fix a defect that has just been discovered than to attempt a repair after further changes have obscured the original cause.

Successful development projects must deliver products that address customer needs to the required level of quality. Linking unit, integration, validation, and verification tests to requirements is fundamental, and automating traceability is key to efficiently execute projects as changes occur and defects are detected.

Model-based testing and the UML Testing Profile

Model-based testing uses modeling to construct and execute the necessary artifacts to perform software testing. The UML Testing Profile[1] extends the applicability of UML to include model-based testing. Using this profile, test architectures can be automatically created for a system from the definition of its interfaces. Test cases consistent with the latest requirements can be defined graphically using sequence diagrams, state charts, or activity diagrams, providing a common modeling language to define test inputs and expected results (see Figure 1). This approach tightens the linkage between requirements, design elements, and tests, thus facilitating better traceability that can benefit both agility and impact analysis. Test cases can be executed on the developer’s desktop and on the target, improving testing productivity.

Strengthening the bond between development and quality management

This move to model-based developer testing doesn’t directly benefit the Quality Assurance (QA) team, as QA engineers typically don’t want, nor should they be forced to become UML specialists. What is needed is a way to allow QA engineers to make use of model-based testing resources without the need to author and own them. This can be achieved through model-driven testing tools that link the tests to the QA test management environment. The tools allow tests to be executed in place from the test management tool and the results to be passed automatically to the test management repository. Tests are then managed in a single location, avoiding the inconsistencies and inefficiencies of duplication, and are available for execution at any point in the development process.

Bringing test result data back into the test management environment can automate the defect resolution process, further optimizing the development and testing process. The QA test management environment should also support linkage to requirements management tooling to ensure that tests can be linked back to specific requirements. This will facilitate test coverage automation, enabling test sets to be automatically defined against requirements or changes.

Taking a project to the next level

By implementing a model-based testing tool connected to test management, embedded development teams can take the following tangible steps to make their projects more successful while freeing up valuable time for innovation:

• Consistency: A central test repository across the development organization will improve efficiency and lead to higher quality. Rather than individual test engineers creating multiple versions of tests through error-prone, manual replication processes, tests are written once and reused as required throughout the development process. A single source of truth for tests can also aid collaboration and ultimately improve delivered quality, as tests are more available throughout the development life cycle, encouraging more frequent testing.
• Communication: Model-based test execution within the quality management environment means that QA engineers can run tests and act upon results without having to be modeling specialists. They can navigate from a failed test to the related requirements and to the source of the problem in the design mode or associated code. This can be particularly useful in distributed and offshore development environments where it can help engender collaboration and build empathy between otherwise isolated developers, testers, and quality professionals.
• Automation: Improved automation of test creation, execution, and results management will significantly reduce the cost and time of testing. This allows more testing to take place, reducing the risk of regression issues in complex projects. In the same vein, automated defect tracking/resolution connects model-based testing to quality management with a backbone of traceability from requirements through to code. This ensures developers have timely and quantitative information to fix defects, and that the effects of defects on delivered functionality are understood. As teams strive to become more agile, it is imperative to prioritize defect resolution over new functionality to avoid accumulating technical debt.
• Agility: Model-based testing conducted using the same modeling notation and tools as design activities facilitates test-driven development. Traceability from requirements to testing will be the key to quantifying results, helping answer the critical question in any project: “Are we ready to ship?”

A call to action

Model-based testing can be considered a first step to bring testing efficiency on par with model-driven development. While its immediate effects are in automating the developer testing environment (with correct linkage to the QA environment), it can enable much wider benefits (see Figure 2). By providing tool support for test execution, test version management, and results management within the QA environment, coupled with life-cycle traceability, a greater degree of test automation can be implemented, eliminating a key bottleneck in the development life cycle.

To realize these benefits, embedded development teams should:

• Extend their modeling capabilities from design to testing with tools that support the UML Testing Profile, such as IBM Rational Rhapsody.
• Ensure their modeling and test management environments are closely linked through tools, such as the IBM Rational solution for real-time and embedded software development.
• Adopt, enforce, and continually improve repeatable processes supported through development and testing tools.

In this way, teams can achieve improved collaboration, productivity, and agility, helping them deliver higher-quality products more rapidly at reduced cost.

References

[1] UML Testing Profile, OMG, June 2011: www.omg.org/spec/UTP/1.1/Beta2/PDF/

Dominic Tavassoli is director of industry and systems marketing at IBM Rational.

Jonathon Chard is systems market manager at IBM Rational.

IBM Rational jon.chard@uk.ibm.com www.ibm.com/software/rational

Follow: Twitter Blog Facebook LinkedIn YouTube

Nearly half of all critical security leaks in embedded software are due to heap overflows. Stack-based buffer overflows account for a smaller percentage, but are exploited with the same technique to inject and execute unauthorized code or change execution flow. Instead of policing such attacks to manage security risk, a better approach is to use the strength of quality software development and code testing with static analysis to find and fix the underlying defects that lead to security vulnerability.

“Because that’s where the money is.”

That’s what Willie Sutton, a prolific bank robber in the United States from the late ’20s to the early ’50s, reputedly said when asked why he robbed banks.

Embedded systems typically don’t suffer from popular sources of Web application security exploits like cross-site scripting and SQL injection. However, security threats are real and ever-present in embedded systems. So where’s the “money” when it comes to security in embedded software?

When reading through the release notes of most software vendors, developers will notice that security patches often contain fixes to a variety of buffer overflow defects. For example, take a look at the security update for Mac OS X at http://support.apple.com/kb/HT4723. It contains a good number of string and heap buffer overflows.

On average, about half of all critical security leaks are caused by heap overflows. Stack-based buffer overflows in embedded software are also a major source of security exploits from altered code. Even without the security risk, buffer overflows are still problematic, as they can cause program execution to halt or produce unexpected values that can be tough to trace at execution time.

With a large amount of software services moving to the cloud, embedded systems at the core of cloud computing infrastructures are more exposed to threats from unauthorized code execution, arbitrary control of resources, corruption of sensitive information, and Denial of Service (DoS) attacks. Developers can benefit from looking at what goes on behind the scenes in a simple buffer overflow and seeing how it allows hackers to change program flow or execution.

What happens in a buffer overflow?

Memory management lies at the heart of buffer overflow exploits. At its most basic level, the problem arises from the fact that Operating Systems (OSs) mix variables and buffers from the program with program execution data. If developers can overflow the program data buffer and overwrite program execution data, they usually can alter program execution.

In its simplest form, a program in memory is organized in three sections/segments: text, data, and stack (Figure 1). The text section, also known as the code segment, is the fixed-size, read-only area that contains code and instructions on executing the program. Similarly, the data section is a fixed-size segment that contains the global variables and static variables initialized in the program.

The stack is the area most relevant to this discussion. It starts at a fixed address, with a register pointing to the top of the stack. Elements called stack frames are PUSHed when calling a function and POPed when returning. A stack frame contains the value of the instruction pointer when the function is called. This instruction pointer is necessary to alter an execution flow from a buffer overflow.

A simple example can illustrate what a stack looks like:

void password (char *buf) { char var[16]; strcpy(var, buf); }

void main () { password(“mypassword”); printf(“This should be executed first\n”);

printf(“This should be executed next\n”); }

When this program is executed, it will show the following:

This should be executed first This should be executed next

The key to understanding what can be done with the stack is to look at the assembler code generated by the compiler. For this discussion, assume this program is executing on an Intel x86 CPU and the OS is Linux.

In main(), it shows:

call password ← This will push the instruction pointer (IP) on to the stack so that it can be used as a return address (RET)

And in password(), it shows:

pushl ebp ← This pushes the frame pointer (EBP) on to the stack movl ebp, esp ← This copies the stack pointer (SP) onto EBP making it the new frame pointer (SFP)

Now when password() is called, the stack looks as depicted in Figure 2.

Analyzing the attack

In lieu of this example, how can the stack be attacked, considering that there is a fixed-sized buffer on the stack that can overflow? Notice that just before the buffer var[16] on the stack is the stack frame pointer, and before that is the return address. So if var[16] could be modified and filled with a value larger than the 16 characters allocated, code could then be executed at the address filled with the return address. In this simplistic example, password() calls an unprotected string-copy function (strcpy()), allowing var[16] to overflow and thus change the return address.

Using GDB can obtain the address of an instruction to execute:

(gdb) disassemble main Dump of assembler code for function main: 0x0804840e <main+0>: lea 0×4(%esp),%ecx 0×08048412 <main+4>: and $0xfffffff0,%esp 0×08048415 <main+7>: pushl -0×4(%ecx) 0×08048418 <main+10>: push %ebp 0×08048419 <main+11>: mov %esp,%ebp 0x0804841b <main+13>: push %ecx 0x0804841c <main+14>: sub $0×4,%esp 0x0804841f <main+17>: movl $0×8048514,(%esp) 0×08048426 <main+24>: call 0x80483f4 <password> 0x0804842b <main+29>: movl $0x804851f,(%esp) 0×08048432 <main+36>: call 0×8048324 <puts@plt> ← Call to first printf() 0×08048437 <main+41>: movl $0x804853d,(%esp) 0x0804843e <main+48>: call 0×8048324 <puts@plt> ← Call to second printf() 0×08048443 <main+53>: add $0×4,%esp 0×08048446 <main+56>: pop %ecx 0×08048447 <main+57>: pop %ebp 0×08048448 <main+58>: lea -0×4(%ecx),%esp 0x0804844b <main+61>: ret End of assembler dump.

Now it’s a matter of passing a string larger than 16 characters that contains the address 0x0804843e, overflowing the buffer and changing the program execution flow.

Sophisticated code testing

Writing embedded code that works, does what it is supposed to do efficiently, and is simple to understand and maintain is not an easy task. While higher-level security concerns like these are often addressed later in the development process, they can be tackled earlier by using proven practices to build quality software.

Defects like string buffer overflows that open possible exploits can be mitigated by not using functions like strcpy() that produce unprotected copies of character arrays to fixed-size buffers. In addition, techniques such as automated code testing using static analysis provide a more sophisticated approach to avoiding static and dynamic buffer overflows.

The following techniques make static analysis an extremely powerful tool for finding programming errors accurately and efficiently at compile time.

Data flow analysis

Modern static analysis tools use data flow analysis to identify the execution path during compile time by creating a control flow graph representation of the source code.

In Example 1, the if statements have four possible execution paths through the code. When the value of x passed into the function is not zero, p is assigned a null pointer with p=0. Then the next conditional check (x != 0) takes a true branch, and p is dereferenced in the next line, leading to a null pointer dereference.

Interprocedural analysis

Another useful technique that static analysis employs is interprocedural analysis for finding defects across function and method boundaries.

Example 2 contains three functions: example_leak(), create_S(), and zero_alloc(). To analyze the code and identify the memory leak, the analysis engine traces the execution to understand that memory is allocated in zero_alloc(), initialized in create_S(), and leaked when variable tmp goes out of scope when it is returned from function example_leak().

False path pruning

A third technique that smart static analysis uses is false path pruning. Regardless of the type of defect or its impact on the embedded system’s security or quality, automated defect-reporting tools must be accurate. This expectation is the same for static analysis; the tool should report critical defects and not false positives. A key to ensuring that the reported defects are real is to only analyze the executable paths.

Example 3 is slightly modified from Example 1. In this case, the execution path simply cannot be executed. Consider the case where the first conditional check (if (x != 0)) results in the false case being evaluated. This will assign variable p the value of 0. At the next conditional check, if the analysis engine looks at the true path it will report a null pointer dereference defect, but that would be a false positive because the execution logic will never traverse this path. It is not possible to evaluate the same conditional check (if (x != 0)) in two different ways. By pruning a path that can never be executed (a false path), good analysis can report up to 50 percent fewer incorrect defects.

Data flow analysis, interprocedural analysis, and false path pruning are only a few of the tricks that a good static analysis tool uses to provide a list of actionable defects that, left unchecked, can have a direct impact on software security … and that’s “where the money is.”

Rutul Dave is senior development manager at Coverity.

Coverity coverity@lewispulse.com www.linkedin.com/company/coverity www.facebook.com/Coverity www.twitter.com/@Coverity www.coverity.com

Despite the widespread misconception that some data races are entirely harmless, modern optimizing compilers generate code that can cause incorrect execution when data races exist. Using static and dynamic analysis, programmers can find and eliminate these defects that are often inadvertently introduced into their code.

Programming multicore processors to take advantage of their power means writing multithreaded code. C and C++ were not designed for concurrency, so developers must use a library such as pthreads for those languages. Multithreaded code is more difficult to get right than single-threaded due to the risk posed by entirely new classes of programming defects.

In the rogue’s gallery of concurrency bugs, the race condition is a notorious repeat offender. A race condition occurs when a program checks a resource property and performs an action on the assumption that the property has not changed, even though an external actor has slipped in and changed that property.

A data race is a particular type of race condition that involves concurrent access to memory locations in a multithreaded program. This defect occurs when there are two or more execution threads that access a shared memory location, at least one thread is changing the data at that location, and there is no explicit mechanism for coordinating access. If a data race occurs, it can leave the program in an inconsistent state.

The insidious nature of data races

It is widely assumed that some data races are harmless and can be safely ignored. Unfortunately, this is only true in very rare circumstances. It is best to explain why by introducing an example.

The singleton pattern is a commonplace idiom where the program maintains a reference to a single underlying object and a Boolean variable encodes it if it has been initialized. This pattern is also known as lazy initialization. The following code is an example of the pattern:

if (!initialized) {

object = create();

       initialized = true;

}

… object …

This code is perfectly adequate for a single-threaded program, but it is not thread-safe because it has a data race on the variable named initialized. If called by two different threads, there is a risk that both threads will observe initialized as false at essentially the same time, and both will call create(), thus violating the singleton property.

To make this thread safe, the natural approach is to protect the entire if statement with a lock. However, acquiring and releasing locks can be costly, so programmers try to avoid the expense by using the double-checked locking idiom – a check outside the scope of the lock and another inside. The inner check is there to confirm that the first check still holds after the lock has been acquired:

if (!initialized) {

       lock();

       if (!initialized) {

            object = create();

            initialized = true;

       }

       unlock();

}

  … object …

Superficially, this looks like it will suffice, and indeed, it will as long as the statements are guaranteed to execute in that order. However, an optimizing compiler might generate code that essentially switches the order of object = create() and initialized = true. After all, there is no explicit dependence between those two statements. In that case, if a second thread enters this code any time after the assignment to initialized, that thread would then use the value of object before it has been initialized.

Optimizing compilers are inscrutable beasts. Those that optimize for speed will take many esoteric considerations into account, few of which are obvious to a programmer. It is common for them to generate instructions that are apparently out of order because doing so might result in fewer cache misses, or because fewer instructions are needed.

It is wrong to assume that because the reordering introduced a race condition in the previous example that the compiler is at fault. The compiler is doing exactly what it is allowed to do. The language specification is perfectly clear and unambiguous about this: The compiler is allowed to assume that there are no data races in the program.

In actuality, the specification is somewhat broader: Compilers are allowed to do anything in the presence of undefined behavior. This is sometimes facetiously referred to as catch fire semantics; the specification gives the compiler permission to set a computer on fire if the program has undefined behavior. As well as data races, many traditional bugs such as buffer overruns, dereferences of invalid addresses, and so on constitute undefined behavior. Because compilers are free to do anything, rather than burn the building down they typically do the sensible thing, which is to assume that the undefined behavior will never happen and optimize accordingly.

The consequences of this can sometimes be surprising, even to those who are experts in concurrency and compilers. It can be difficult to convince programmers that code that looks completely correct can be compiled into code that has serious errors.

Another example is worth describing. Suppose there are two threads where one reads a shared variable and the other writes to it. Let’s assume that it does not matter to the reader if it sees the value before or after it has been changed by the writer (this is not an uncommon pattern). If those accesses are not protected by a lock, then there is clearly a data race. Notwithstanding the catch fire rule, however, most programmers would conclude that this is completely benign.

As it turns out, there are at least two plausible ways in which this code could be compiled where the reader would see a value that is wrong. The first way is easy to explain: Suppose the value were a 64-bit quantity on an architecture that can only read 32-bit words. Then both the reader and the writer need two instructions, and an unlucky interleaving might mean that the reader sees the top 32 bits of the old value and the bottom 32 bits of the new, which when combined can be a value that is neither the old nor the new.

The second way in which wrong code could be generated is more subtle. Suppose the reader did the following, where the data race is on the variable named global:

int local = global;  // Take a copy of
                       // the global

if (local == something) {

       …

}

… // Some non-trivial code that does
      // not change global or local

if (local == something) {

       …

}

Here the reader is making a local copy of the racy variable and referring to that value twice. It is reasonable to expect that the value will be the same in both places, but again, the optimizing compiler can generate code where that expectation is unmet. If local is assigned to a register then it will have one value for the purposes of the first comparison, but if the code between the two conditionals is sufficiently nontrivial then that register may get spilled – in other words, reused for a different purpose. In that case, at the second conditional, the value of local will be reloaded from the global variable into the register, by which time the writer might have changed it to a different value.

Programmers should be very skeptical of claims that some data races are acceptable and should strive to find and remove all of them from their code.

Techniques for finding risky defects

When it comes to finding concurrency defects, traditional dynamic testing techniques might be inadequate. A program that passes a test a hundred times is not guaranteed to pass the next time, even with the same inputs and the same environment. Whether these bugs manifest or not is exquisitely sensitive to the timing, and the order in which operations in threads are interleaved is essentially nondeterministic.

New dynamic testing techniques for finding data races are emerging. These techniques work by monitoring the applications as they execute and observing the locks held by each thread, as well as the memory locations being accessed by those threads. If an anomaly is found, then a diagnostic is issued. Other tools help diagnose data races suspected of causing failures. Some companies now offer tools to facilitate diagnosis of data races that allow the replay of events leading up to an anomaly.

Static analysis tools can also be useful for finding data races and other concurrency errors. Whereas dynamic testing tools find defects that occur for particular executions of a program with a fixed set of inputs, static analysis tools check all possible executions and all possible inputs. For performance reasons, tools might place limits on how much exploration is done and thus might not be completely exhaustive; even so, they can cover much more than can ever be feasible with dynamic testing. The advantage of static analysis is that test cases are not required because the program is never actually executed.

Instead, these tools work by creating a model of the program and then exploring the model in various ways to find anomalies. GrammaTech’s CodeSonar finds data races by creating a model that represents the set of locks held by each thread and by performing a symbolic execution of the program that explores execution paths. It records the sets of variables protected by locks and uses this information to find interleavings that can result in shared variables being used without proper synchronization. Similar techniques can be used to find other concurrency defects such as deadlock and lock mismanagement.

Once found, data races are usually easy to fix, albeit doing so correctly can incur a performance penalty. In some cases, there might be a temptation to use the C volatile keyword to correct the data race, but this is not recommended because volatile was not designed to solve concurrency problems, and in any case is a poorly understood construct that is frequently miscompiled. The latest versions of C and C++ have embraced concurrency and support atomic operations. Compiler support for these operations is slowly emerging, and until it becomes readily available, the best approach is to use locks.

To achieve high-quality software for multicore processors, a zero-tolerance policy for data races is recommended. Find them using a combination of both static and dynamic techniques, and take care not to rely too heavily on esoteric compiler techniques to fix them. These defects are so risky and unpredictable that eliminating them systematically is the only safe way to be sure that they do not cause harm.

Paul Anderson is VP of Engineering at GrammaTech.

GrammaTech
607-273-7340
paul@grammatech.com
www.grammatech.com

The ubiquitous nature of embedded software has made source code analysis a critical component of the development process. Gwyn discusses the impact this technology has on software security and reliability, and emphasizes the importance of making static analysis a natural part of a developer’s coding practice.

ECD: As embedded developers turn to multicore processors to optimize performance, how can analysis tools help control inevitable cost and schedule problems?

FISHER: Any new development is an exercise in balancing expectation against risk. In the case of multicore, the naive expectation is always linear acceleration tempered perhaps by some jocular “wouldn’t that be nice” acceptance that the final result won’t be quite that good, but no real understanding of the reality that without significant effort (read: time, money, angst) the result might be slower than the old, interrupt-driven single-core code. So tools have a role to play in terms of helping developers understand the impact of what they’re doing, what pitfalls they’re unwittingly leaving themselves open to, and how to mitigate the associated risks.

Dynamic analysis in this space has received the lion’s share of attention for obvious reasons. If I can see a nice set of graphs over time that shows the performance of my code in action, I can presumably narrow in on problem areas quickly and apply my own knowledge to figuring out what’s going on. The challenge with dynamic analysis is that it depends on a) defining a test set that shows execution problems, and b) the reviewer’s intimate knowledge and understanding of what to do about those problems.

Static analysis, by contrast, assumes little knowledge on behalf of the reviewer and requires no effort to be expended in defining test cases. Every conceivable code path through the application is exercised with as much rigor as every other path. This approach is therefore far more likely to show complex and costly issues such as data races, deadlocks, and resource contention than any constructed test bench. That speaks directly to the bottom line of cost control in what is inevitably a large, over-budget project. Leaving issues such as these in a code base until late in the validation process will cost exponentially more to address than doing so during initial development.

In addition, due to the way that static analysis works by modeling the expected program execution, the finding is accompanied by a detailed walk-through of how the situation is predicted to occur, allowing even a relatively junior resource to interpret the issue at hand, determine whether or not it’s likely to happen, and apply an appropriate design fix. In one example we like to describe in seminars on the subject, a design flaw in a popular open-source database kernel resulted in months of effort expended to identify a deadlock and eventually rewrite key modules to avoid the data race at its heart. This same problem was identified during the first analysis using our tools, which provided a walk-through description enabling developers to easily see that the data race was causing the problem and that the deadlock was merely the symptom.

Contrast a few hours to run the tool to analyze the code and an hour at most to interpret and act upon the result (what turns out to be a one-line fix) with months of community effort to determine an appropriate set of tests, followed by design effort attempting to fix the deadlock, and finally requiring the designer to rewrite the whole thing from scratch.

ECD: How is static analysis introduced in the software development cycle, and how can it be used with existing Integrated Development Environment (IDE) tools?

FISHER: There’s absolutely no doubt that any developer-facing tool that doesn’t integrate seamlessly with any existing tooling is going to face significant friction in deployment. We’ve been selling this message effectively for years, with development managers almost asking the “why wouldn’t you do it this way?” question for us.

Whether developers have migrated to IDEs or their idea of an IDE is a bunch of gVim macros or emacs lisp modules, to them it’s where they live and work. And woe betide any vendor who tries to get them to change. Even if you’re not suggesting that they change tools, and instead asking them to visit somewhere else to see what they might have done wrong in some retrospective manner, your tool is going to suffer waves of disinterest and ultimately become shelfware.

Thus, static analysis has to be part of the developer’s native habitat, and more importantly, it has to work in a way that feels natural and follows the way other tools in that habitat work. For a gVim developer, issuing a ‘:’ command is second nature, so tools in that environment should follow suit. Put that same interaction mechanism in front of a Visual Studio user, and that will make for a fun-filled afternoon of derisive commentary.

At Klocwork we’ve gone through various iterations of technology design, getting closer to the developers themselves. It’s one thing to be resident as a tool within an IDE; it’s another to get the developer to “push the button” to use it. Making a button available is only changing geography, and it does nothing to help with the tool’s fundamental usability.

With this in mind, we’ve recently introduced a new technology that allows static analysis to take place in much the same way as spell checking in a Word document or e-mail. That is, as you’re writing your code and the tool detects a problem with what you’re doing, it can point out the problem in a perceptually instant manner, highlight it with a squiggly underline, and deliver all the value of static analysis with none of the “what do I have to do to use it?” resistance that is typically encountered during any new tool’s introduction (see Figure 1).

Getting a tool into an IDE is tough; getting it to be useful in the part of the IDE the developer truly lives in (the editor component, whatever that looks like in the environment at hand) is really tough. But until you’re there, you’re a distraction.

ECD: Can source code analysis be used to protect embedded devices against potential security threats?

FISHER: Absolutely, source code analysis has a significant role to play in threat identification and validating whatever threat model is being used to determine vulnerability in the device. The connectivity requirement is common in the embedded world today. That connection might be to another chip, or another device, or the whole Internet. In any case, there’s somebody else either sending you information or receiving information you’re sending. That’s the starting point for having to worry about your entire application design.

Static analysis doesn’t typically know, or try to know, anything about your surrounding environment. Tools can sometimes be tuned to perform their analysis within certain data boundaries, for example, such as knowing that a particular input will only range between -20 and +30 because it’s a temperature sensor intended for use in Western Europe. But that kind of thing is discouraged because you’re allowing the user to define limits to what the modeling technology naturally does – that is, assume nothing and point out everything that looks wrong.

In the case of threat detection, we’re most worried about how the data you’re interacting with from the outside world is used internally. Is it used to create a buffer into which you’ll read data (code or value injection), or is it used for memory allocation (Denial of Service or DoS), or perhaps interpreted as a reference into an internal data structure (hijacking or redirection)? This kind of data and path validation – that is, the path that tainted data follows from the outside world to its point of use within the code – is natural for source code analysis, as it accomplishes this modeling in order to perform everything else it does.

As a wonderful gentleman at a defense contractor once said to me, “Son, it’s a bomb; it’s supposed to blow up. We just want to be sure it doesn’t get hijacked on its way.”

ECD: What educational events or online classes does Klocwork offer to help embedded designers get started with its code analysis tools?

FISHER: Like any commercial organization attempting to encourage users to gain value from its tools, Klocwork provides a full suite of educational and professional services, running the gamut from introductory materials aimed at first-time users, to more advanced courses targeting secure coding and threat modeling, to full-on deployment services and mentoring.

We also recently introduced the Klocwork Developer Network (http://developer.klocwork.com), a website serving our users and acting as a repository for online courses, video tutorials, in-depth courseware, and the usual variety of community forums and ticketing.

Each customer has something unique they wish to gain from a tool such as source code analysis, so a large part of our educational focus is on internal champions, people who take knowledge of how the tools work and how other organizations have applied them and leverage those lessons in deploying our tools for their own use. A large part of that is learning from the community, so I’ve been thrilled to see the fast uptake that the Klocwork Developer Network has seen amongst users, both as a less formal mechanism for interacting with our staff, but most importantly as a way to learn from other customers.

Gwyn Fisher is CTO of Klocwork.

Klocwork info@klocwork.com www.klocwork.com/blog www.facebook.com/klocwork www.twitter.com/@klocwork www.klocwork.com

As more and more embedded devices evolve from single-function controllers to complex platforms supporting high-speed graphics, user interfaces, and network communications in addition to the primary application, real-time responsiveness is becoming a critical performance requirement. Although developing in-house software offers some advantages, the benefits of reduced complexity and shorter development schedules often justify the purchase of a commercial Real-Time Operating System.

The average person interacts with hundreds of embedded processors every day in phones, automobiles, home appliances, toys, cash registers, entertainment electronics, security systems, environmental controls, and personal electronics. The common link among all of these products is their ability to react in real time to the user, external events, and the communications channel.

The software for these embedded devices can be divided into application software and Operating System (OS) software. Application software makes the product unique and contains the data collection, signal processing, and hardware control routines required to make the product perform to its specification. The OS allows the programmer to break up large application programs into smaller, individually developed processes or tasks.

At the heart of an OS is the kernel, which schedules programs for execution and manages shared resources. A Real-Time OS (RTOS) processes hardware requests or interrupts from timers or external events within a guaranteed maximum time. Programmers interact with the OS through an API and set up the priorities and data dependencies. During execution, the RTOS manages the application software with a flurry of external real-time activity.

In-house code

Even with the advantages of an RTOS, homegrown OSs still occupy a non-trivial percentage of embedded real-time products. Developers have multiple incentives for bypassing a commercial RTOS entirely and writing their own real-time routines. The biggest reason developers cite for not choosing a commercial OS is lack of need. With only one task running, designers think they can easily keep track of the required hardware interaction.

Special situations sometimes justify in-house software. For example, the design objectives of a portable health care device can include low cost, low power, and a one-year battery standby life without extra memory and processing power to support a commercial RTOS. Furthermore, if a new project is an upgrade of a previous project, developers likely will want to use as much legacy code as possible.

Components that aren’t invented at the same company might also be one reason why many developers write their own OSs. Installing software from a third party into their showpiece product is like admitting they are somehow not up to the task. In addition, developers might think they’ll lose the ability to make software adjustments to compensate for hardware changes or to correct bugs. The designer can easily adjust the order of execution or drop to assembly language to solve critical timing problems with in-house developed software. However, with a commercial RTOS, the scheduler handles many of the timing issues, so developers lose the perception of being in total control. And finally, programmers list sticker shock as another reason to write their own operating software. The initial license for a full commercial RTOS and associated tools can be in the $15,000 to $20,000 range for a single development seat, plus recurring royalties for every unit shipped.

Software shortcuts

As embedded systems grow in complexity and project schedules shrink, software has displaced hardware as the highest-priced item in most embedded development projects. If design teams can buy an RTOS and eliminate the coding, debug, and documentation of the most complicated portion of the software structure, then the purchase decision should receive careful consideration. Although a commercial RTOS can be expensive, a smaller development team and shorter project time frame might create more than enough savings to justify the purchase.

An RTOS allows programmers to write independent, reusable modules to reduce software complexity and shorten the development schedule. Programmers can write each software routine independently without getting bogged down with intertask timing problems. Most RTOS vendors provide a full interactive development environment including a source code editor, code manager, linker, downloader, runtime tools, and one or more debuggers. Software vendors also supply software performance analysis tools to help profile and visualize real-time activity in application routines. Programmers can monitor which tasks are running, observe the stream of data flow, and detect when and how often a task is interrupted by a higher-priority item. RTOS vendors agree that high-quality development tools can dramatically shorten debug time.

Along with the cost savings, RTOS vendors cite multiple technical reasons to justify their products. For example, if an application involves heavy data processing, many RTOSs can be scaled easily to spread tasks across several processors for a significant performance boost. The RTOS provides communication and synchronization services to make multiprocessing transparent. In addition, an off-the-shelf RTOS working alongside multicore processors simplifies legacy code integration within new designs or products updates.

A commercial RTOS is modular, so users can select only those portions or features of the OS that they need. Specifying a subset of the full-blown commercial RTOS can reduce acquisition costs and the required memory footprint. With the current connectivity trend, even the simplest embedded products might need to connect to and send data over the Internet. A graphical user interface could also become standard in small embedded systems, even if just for maintenance. These features are included or optionally available in most commercial RTOSs, but can be very expensive or impossible to add to a proprietary OS. Vendors also promote product on-demand technical support as a major benefit of a commercial RTOS.

Off-the-shelf platforms

Commercial RTOSs are constantly upgraded to add new features and keep up with changing technology. For example, the popular VxWorks OS from Wind River was recently revised to deliver 64-bit computing support along with improved multicore features. VxWorks includes a shell, debugging functions, memory management, performance monitoring, and support for multiprocessing. Real-time features include a kernel for preemptive multitasking, interrupt response, interprocess communication, and a file system (see block diagram in Figure 1). Software development is enabled by the Wind River Workbench development tools suite and Intel Integrated Performance Primitives for VxWorks.

The RTOS supports various multicore configurations in Symmetrical Multi-Processing (SMP) and Asymmetrical Multi-Processing (AMP) modes or as a guest OS on top of Wind River Hypervisor. VxWorks also has a configurable and tunable small memory footprint, allowing the user to control how much of the OS to employ for each project.

In addition to offering a multitude of commercial RTOS products, the embedded systems community maintains an open-source OS based on a real-time kernel that is free for use in commercial applications. The FreeRTOS Project is under continuous active development and is distributed under the GNU General Public License with an optional exception that allows users to keep their proprietary software confidential. Free source code and the lack of recurring royalties are popular features for small, low-budget embedded projects. FreeRTOS has been ported to multiple microcontroller platforms and has minimal ROM, RAM, and processing overhead, resulting in a typical kernel binary image in the 4 KB to 9 KB range. Although FreeRTOS source code for the kernel is contained in only three C code files, the zip file download includes numerous demonstration applications to help new users get started.

The biggest complaint among potential open-source software users is the lack of a central resource to provide support similar to that offered by a commercial software vendor; however, the FreeRTOS website has an active free support forum where developers can find answers to their technical questions. In support of the open-source platform, Microchip Technology offers the FreeRTOS Microchip PIC32 Education Kit (see Figure 2). This $95 kit includes a development board that enables users to develop USB embedded host, device, and On-The-Go applications on the PIC32 microcontroller family.

Real-time future

Although programmers might get excited when considering the challenge of developing an in-house OS, the “roll your own” days might be fading away. Designers can look forward to real-time software as the norm in future embedded products.

Customer demand for faster response times, complex functionality, and instant data access continues to increase the challenge of embedded design. Advancing technology also dictates that embedded products be capable of periodic software updates as requirements change, along with the possible transfer to the next-generation hardware platform.

Developers should take the time to analyze their system requirements, development schedule, software support, expandability, communications, scalability, and future growth before embarking on an in-house software development project. An off-the-shelf commercial RTOS or even an open-source operating system could be in your future.

Companies today are seeking to own more of the vertical design chain by bringing chip design in-house. If not done properly, this will create more problems than it solves. The key is to use a common high-level model of the hardware for software development and debugging that can also be taken into an automated hardware implementation flow. By developing hardware models in SystemC-based Transaction-Level Modeling (TLM), software teams can debug against virtual prototypes long before a hardware prototype is available.

The consumer and wireless communications markets are more competitive than ever. The ongoing battle between aggregation and disaggregation of companies is in full swing. One example of aggregation is a decision to own more of the vertical design chain by bringing chip design in-house. This has helped companies like Apple differentiate by controlling more of the overall product design and thus not being limited by off-the-shelf chips that are available to everyone else.

While Apple has demonstrated the potential reward of vertical differentiation, this approach does pose significant risk, whether a company has experience designing chips or not. Specifically, how does the software team develop software that works with the hardware as shipped?

On the other side of the equation, complete disaggregation is enabled by software abstraction layers like Google’s Android operating system. It has somewhat democratized the design space, allowing all system companies to participate and differentiate using software. Android allows semiconductor vendors to participate equally by providing supporting hardware. Again, the way the software works with the hardware determines product success.

Traditional solutions to this problem do not work in today’s market. Companies used to be able to start software development based on specifications and wait for chip prototypes to be available for testing. That works if the software is very simple, independent of hardware, and has a straightforward specification, but not with today’s consumer electronics that require everything be connected.

Furthermore, waiting a long time to begin testing pushes the debug cycle out far too late in the schedule. Many companies addressed this in recent years by moving to standard off-the-shelf chips, but that approach limits the ability to differentiate. What if you want to add a power-saving sleep mode but there’s no way to shut down the chip?

In an aggregated scenario, companies are looking to differentiate not only in software and industrial design, but also in electronic hardware. Embarking on a chip design project poses its own risks; couple that with embedded software development, and overall project risks go up exponentially. Most companies are careful enough to spend ample time up front architecting the system, testing it, partitioning it into software and hardware, and specifying the behavior of both. But once each team begins designing, certain implementation assumptions are made, bugs are introduced, and features can be added.

The scenario is even worse in a disaggregated world, as the responsibility now spans across company borders. Companies from the systems and semiconductor world might decide to work together to optimize hardware/software interaction and create a chip optimized for system needs. Even if there are constant synchronization meetings, design changes will sneak in unbeknownst to the software team and might not be seen until the first time the software is run on the actual hardware. This cycles back to the problem of the hardware not being available soon enough. How do engineers solve this conundrum?

A golden model for prototyping

Virtual prototypes (or virtual platforms) of hardware that come in the form of software models give the software team a model of system hardware earlier in the process. This enables developers to begin testing on a model of the hardware specification. However, it is only a model of the specification. Most hardware design today starts with engineers reading and interpreting the specification, then writing low-level Register-Transfer Language (RTL) models in a hardware design language such as Verilog to begin the verification and implementation process. Due to the factors mentioned previously, hardware behavior will likely diverge from the specification.

The solution is to use a common “golden model” on which the software team can develop and with which the hardware team can begin their implementation. This is now possible with the availability of the Open SystemC Initiative (OSCI) Transaction-Level Modeling (TLM) 2.0 standard.

In short, SystemC is a class library that enables hardware design using C/C++ by modeling hardware data types and concurrency. Because hardware can now be modeled in C, that same model can be run by the software team. The TLM extensions are important because they abstract away all the signal-level protocol details the hardware needs to ensure that it communicates properly with the system bus. An excess of these details makes the model too slow for running the software. TLM abstracts those details to higher-level models that can be mapped to detailed hardware during high-level synthesis.

Resolutions to high-level synthesis limitations

High-level synthesis provides the automated link between the C model and the actual hardware that gets built. This removes the human factor of the hardware designers interpreting the specification and manually writing their own model to begin building the hardware. Until recently, this had rarely been used in practice because of some key limitations that have now been addressed:

• Quality of results: The first two generations of high-level synthesis were never able to produce hardware that met the same performance, power consumption, and size that could be achieved by manually writing RTL. Modern high-level synthesis technology has resolved this issue.
• Refinement methodology: The high-level virtual prototype for software development is described with SystemC TLM, but it still requires that the hardware team refine it by adding in hardware architecture details so that high-level synthesis can produce optimal hardware microarchitectures. These details are too low-level for software testing and would slow down its speed, but they are important for building efficient hardware. This methodology now exists and has been proven by early adopter customers.
• Verification: Until very recently, engineers lacked a mature methodology to verify the correctness of the hardware architecture in SystemC TLM and the rest of the hardware implementation flow. This is mainly because an automated path into implementation did not exist, so most verification was done at lower levels. Thus verification became the bottleneck in the hardware development schedule. Now that the automated path exists, verification methodology has been developed.

Hardware design teams are familiar with these traditional barriers to designing and verifying hardware using SystemC TLM. Most, however, are not aware that these barriers have been addressed. Those who are aware now enjoy a significant competitive advantage. They can describe their hardware much more efficiently, verify it more rapidly, and reuse it in derivative chips more easily.

Virtual platform in practice

A common model of the hardware is now available as part of a virtual platform much earlier so hardware/software interactions can be addressed sooner. This common model can be delivered as part of the bigger system in a virtual platform either within the company in an aggregated development scenario or across companies in a disaggregated world.

One example of the way this works in practice is captured in Figure 1, which illustrates the flow provided by the Cadence System Development Suite, an integrated set of hardware/software development platforms.

The system concept is first described as a SystemC TLM virtual prototype. In the Cadence flow, this virtual prototype is used by the Virtual System Platform to run the software on this hardware model. In parallel, the hardware design team will refine the TLM to add hardware architecture details for C-to-Silicon Compiler high-level synthesis, which is the beginning of the implementation process that will lead to silicon.

If bugs are uncovered during testing, the Virtual System Platform is integrated with the Incisive Verification Platform so that debug can happen on both software and hardware. This means that issues can be addressed at their source without cumbersome firmware patches. As the hardware implementation process progresses, more detailed RTL models become available to create hardware emulation models in the Verification Computing Platform or an FPGA prototype in the Rapid Prototyping Platform.

This entire process is a successive set of refinements that begins with a fast TLM model, adding more hardware detail as it becomes available while maintaining runtimes that are fast enough for software development. This ultimately enables the software and hardware teams – even across company borders – to have a common model that enables earlier communication and constant synchronization. This is the type of collaboration needed to keep pace with the innovation and delivery schedules required in today’s consumer market. It is only achievable if the hardware team evolves its design and verification methodology to encompass SystemC TLM.

Michael (Mac) McNamara is VP and general manager of system-level design at Cadence Design Systems. In the early 1990s, he helped start Chronologic, which brought compiled Verilog simulation to the world; thereafter, he cofounded SureFire Verification (which became part of Verisity) to improve the state of verification software. After Cadence acquired Verisity, Mac led the effort to improve high-level design, and currently manages Cadence’s C-to-Silicon Compiler and Virtual System Platform product lines.

Cadence Design Systems 408-348-7025 • mcnamara@cadence.com Linkedin: www.linkedin.com/company/cadence-design-systems Facebook: www.facebook.com/pages/Cadence-Design-Systems-Inc/66598923031 Twitter: @Cadence www.cadence.com

In embedded applications, virtualization software can be used to combine a real-time deterministic operating system with a high-level interactive operating system like Windows or Linux. Using virtualization platforms and tools such as those mentioned in the following discussion simplifies system upgrades and optimizes performance by independently allocating system resources to each operating environment.

As embedded technology and market expectations evolve, design engineers are constantly pressured to pack expanded functionality into smaller, reduced-power devices. In addition to the added complexity of the application software for these new projects, customers demand an interactive interface, ubiquitous connectivity, absolute security, and extreme reliability.

Embedded designers also face the challenge of combining slower legacy interface circuitry with the latest high-speed control devices and multiple displays. The resulting system often includes the original hardware with its Operating System (OS) and application software, plus a completely separate controller with software to handle the newer requirements. This approach increases component count and power requirements and does nothing to increase legacy application performance.

To deal with this increased complexity, designers are utilizing virtual processors hosting multiple OSs to ensure unimpeded, deterministic response to real-time events while simultaneously providing users and operators with a high-level, graphics-based interface. Virtualization is achieved by adding a Virtual Machine Monitor (VMM) software layer or hypervisor that isolates individual partitions and executes guest operating software. The hypervisor creates one or more simulated computer environments or virtual machines that can simultaneously host independent OSs and applications on a single processor.

To speed up virtual component interaction, silicon manufacturers are incorporating hardware-assisted virtualization in processor architectures tailored for extended life-cycle embedded applications. For example, the second-generation Intel Core and Intel Atom E6xx processors support Intel Virtualization Technology (Intel VT). This technology improves software-based virtualization performance and security by using hardware assist to trap and execute certain VMM instructions. Intel VT allows the VMM to allocate memory and I/O devices to specific partitions, thus decreasing the processor load and reducing virtual machine switching times.

Virtual isolation

Virtual platforms that combine real-time or safety-critical embedded functions with a large graphics-based OS must contain security provisions that allow unaffected partitions to continue operation in the event of a software failure or cyber attack. For example, LynuxWorks updated the LynxSecure separation kernel and hypervisor for various virtual machine configurations, as shown in Figure 1. This virtualization software is designed to operate in secure defense environments where data and applications with different security levels must co-reside on a single device without corruption. LynxSecure uses a hypervisor to create a virtualization layer that maps physical system resources to each guest OS, which is assigned dedicated resources such as memory, CPU time, and I/O peripherals. Another key feature is the ability to run fully virtualized 64-bit guest OSs such as Windows 7, Linux, and Solaris across multiple cores.

TenAsys Corporation offers eVM for Windows, another embedded virtualization platform that hosts an embedded OS or Real-Time OS (RTOS) alongside Windows on the same processor platform. To ensure that critical hardware interfaces are not virtualized, eVM partitions the platform, thus guaranteeing maximum performance and deterministic response to real-time events. Installed as a standard Windows application, eVM includes all of the integration tools needed to set up, start, and stop multiple RTOS guest configurations. The Windows-based control panel also allows users to assign interrupts, allocate I/O devices, and set up disk boot images. After the system is set up, eVM provides the guest RTOS with the lowest possible interrupt latency, direct access to I/O, and non-paged RAM.

Multicore virtualization

Although virtualization allows designers to combine OSs and applications to reduce system power requirements and form factors, it does little to increase the performance of individual software components. One of the latest trends among designers is to incorporate multicore processors along with virtualization to boost performance through parallel processing.

With virtualization, the hypervisor isolates and allocates system resources between operating environments so that real-time, general-purpose, and legacy software can be readily integrated in a multicore system. In addition to memory and hardware device allocation, virtualization allows developers to assign multiple cores to compute-intensive applications as needed to maximize overall system performance.

Extending virtualization to multicore applications, the Wind River Hypervisor allows designers to configure and partition hardware devices, memory, and cores into virtual boards, each with its own OS, while maintaining the necessary separation (see Figure 2). These virtual boards can be run on a single processor core or distributed across multiple cores based on system needs. The Wind River Hypervisor has been applied in safety-critical applications where the system’s safety-certified and noncertified components traditionally had to be physically separated. However, embedded virtualization allows system designers to isolate the safety-certified components while still operating on a single hardware platform utilizing a certified hypervisor. Virtualization also improves the potential uptime of embedded applications by enabling individual partitions to be rebooted or even reprogrammed while other services on the same device are not affected.

Real-Time Systems also provides virtualization support for multicore processors. Leveraging Intel VT for security, the RTS Real-Time Hypervisor allows completely independent execution of more than one OS on a single multicore platform. Designers can assign individual processor cores, memory, and devices to each OS. Through a configuration file, the boot sequence can be specified, and when desired, an operating system can be rebooted independently of the others. To facilitate communication between OSs, the hypervisor also provides configurable user-shared memory, as well as a TCP/IP-based virtual network driver. The system can run multiple instances of RTOSs mixed with high-level operating software such as Windows XP/CE/7/Embedded, QNX, Linux, On Time RTOS-32, VxWorks, Microware OS-9, and Android.

Development and debug

No matter if virtual applications run on a single processor or across multiple cores, software development and debug tools must be configured to support more than one OS and memory partition. For example, Green Hills Software updated its INTEGRITY RTOS and MULTI Integrated Development Environment (IDE) to support the latest virtualization microarchitecture. INTEGRITY RTOS is built around a partitioning architecture to provide embedded systems with enhanced reliability, security, and real-time performance. Secure partitions guarantee each task the resources it needs to protect the OS and user tasks from errant and malicious code. INTEGRITY architecture provides Asymmetrical Multi-Processing (AMP) and Symmetrical Multi-Processing (SMP) support optimized for embedded and real-time multicore processors.

MULTI IDE software tools include several C compiler options, a debugger, editor, configuration manager, code browser, and debugger in a single package. MULTI also features DoubleCheck, an integrated static analyzer that isolates bugs caused by complex interactions between code segments that might not be in the same source file. In addition, Green Hills Probe provides a multicore debug control for board bring-up, device driver development, and system-level debugging.

The next step is to incorporate multicore support by updating and streamlining the software development tool set while minimizing modifications to current code creation practices. Various software vendors provide advanced development tools and board support packages for products based on second-generation Intel Core devices. For example, the Prism software analysis tool from CriticalBlue (Figure 3) allows developers to analyze existing software applications, evaluate benefits of the new architecture, and select the appropriate processor.

Prism analyzes the behavior of existing code running on simulators or hardware development boards to assess opportunities that introduce or add further parallel code structures. For example, developers can select the appropriate member of the second-generation Intel Core processor family and analyze the impact of Intel Hyper-Threading Technology, data cache misses, and instruction throughput. Prism gives developers an estimate of the performance gain achievable by partitioning the program into multiple threads.

Design simplified, performance optimized

Virtualization is a proven way to simplify embedded designs with fewer components while integrating the framework needed to easily combine disparate operating software or future updates. Virtualization also simplifies system upgrades by isolating the hardware and software layers so that designers can easily add or modify peripherals, memory, and cores without restructuring the software architecture. A virtual machine hypervisor enables designers to optimize performance by tweaking resource mapping even after deployment.

Engineers are turning to verification and validation methods that allow them to find errors in their design and the code as early as possible. Applying these methods gives engineers confidence that their processes are robust in typical production projects where exhaustive testing is neither practical nor possible due to application complexity. Engineers can achieve early verification by maximizing the benefits of Model-Based Design.

Model-Based Design (MBD) performs verification and validation through testing in simulation. Although many organizations use some form of modeling, too many apply simulation in an ad hoc manner that does not take advantage of the potential verification benefits (see Figure 1).

To maximize the benefits of MBD, successful organizations have implemented four key practices that help accomplish early verification:

·    Create and simulate a high-level system model during the specification stage. In MBD, the system model serves as an executable specification. Early simulations of this model highlight incomplete and inconsistent requirements and specifications.

·    Test from day one with multidomain simulations. By developing multidomain models and performing closed-loop simulations, engineers can start testing while the product idea is taking shape. These simulations enable engineers to investigate all aspects of the system, including algorithms, components, the plant model, and the environment.

·    Create virtual test suites that stress the system. Simulations enable engineers to conduct a range of tests that would be difficult or impossible to perform on the embedded system itself. Like all tests, these should be run as early as possible.

·    Use the model and test suites as a reference design throughout the development process. Well-constructed models can be used throughout development and then reused for future enhancements and derivative designs.[1]

These practices should be employed in parallel as four dimensions to successful usage of MBD for early verification.

Create and simulate a high-level system model

To serve as an executable specification, a high-level system model must mirror the system’s abstract behavior. The model might not include the full interface definition, but it must specify the system’s dynamic behavior. Simulating system behavior in the requirements specification stage helps ensure that the team has a complete and shared understanding of what the system is required to do.

With MBD, engineers start by assembling the architecture using either subsystems or discrete states. The dynamics within these subsystems should initially be modeled using the easiest possible approach. In parallel with this activity, other engineers can create scenarios or formalized requirements in preparation for testing the dynamics as early as possible.

When the first tests are run, the engineers who modeled the functional behavior will learn more about the system and the real meaning of the requirements. Likewise, the engineers who created the test scenarios or formalized requirements will learn whether the requirements are consistent and complete. Each group should communicate their findings to the other to make sure there are no misunderstandings.

Test from day one with multidomain simulations

System behavior is defined not only by the embedded control software, but also by the electronic and mechanical components, including the connected sensors and actuators. Early simulations in which the architecture is executed provide more insight when they are performed in a closed loop with plant or environment models.

Closed-loop simulations with plant models offer several advantages over open-loop simulations or testing on actual plant hardware. One advantage is that models are easier to change than metal, wires, and C code. Closed-loop simulations with plant and environment models reduce costs in multiple phases of development. A model is easier to reconfigure and replicate than a mechanical and electrical device built from steel, wires, circuits, and other hardware. Engineers can rapidly switch between versions of the physical model without incurring manufacturing costs. By simply changing parameters such as the length of a rod or the maximum torque of an electric drive, teams can evaluate trade-offs and optimize the complete system for cost, speed, power, and other requirements.

System-level optimization requires multidomain simulations. It is impossible to optimize today’s sophisticated systems by tuning one parameter at a time. To deliver maximum energy efficiency and highest performance at minimal material cost, engineers must optimize the system as a whole, and not just the embedded software.

Plant models provide another perspective on the system. Modeling the nonsoftware parts of the system gives engineers another view into system behavior. Engineers can often learn more about system dynamics through simulation than from the real system because simulation provides details on force, torque, current, and other values that are difficult or impossible to measure on the actual hardware.

Creating plant models requires engineering effort, but this effort is often overestimated, while the value provided by plant modeling is underestimated. When developing plant models, it is a best practice to start at a high level of abstraction and add details as needed. Choosing a level of abstraction that is just detailed enough to produce the needed results saves modeling effort as well as simulation time (see Figure 2).

Create virtual test suites that stress the system

Efficient testing requires a separation of concerns. Organizations should test different aspects of the software implementation at different stages of development. Testing communications and hardware effects before the algorithms have been tested makes it difficult to isolate and identify the source of defects in the design.

Applying tests where and when they are most appropriate enables teams to evaluate the design at the right level for each phase of development. At each phase, test results should be fed back to development immediately to enable continued refinement of the design.

Functional testing involves simulating the controller model with the multidomain environment model. Test vectors used in functional testing are based on either formalized requirements[1] or scenarios such as recorded driving maneuvers. These test vectors can be reused for regression testing and full model coverage testing.

Rapid Control Prototyping (RCP) adds real-time verification and the user experience to the test regimen. RCP helps engineers quickly deploy algorithms and test them in the vehicle to determine whether the functionality feels right. Enabled by on-target rapid prototyping and functional rapid prototyping platforms, RCP can be a rich source of design ideas, but it should not serve as the primary method of verifying functionality.

Robustness testing aims to evaluate system robustness amid changes in software parameters, manufacturing process variances, mechanical and electrical hardware degeneration over the system’s lifetime, and similar effects. It is a best practice to run parameter sweeps on the virtual system, including the controller and environment. With a more thorough understanding of how the system performs at boundary conditions, engineers can choose to narrow the specification for their hardware vendors or conclude that a less expensive part with slightly higher variances meets their design needs.[2]

Hardware-In-the-Loop (HIL) testing enables engineers to test the real controller or controller networks in the lab rather than in a real-world environment. HIL testing can be used to test robustness (for example, by inserting failures) or diagnose intercontroller communication in large controller networks. It covers hardware and communication effects that cannot be modeled easily.

Like RCP, HIL testing is necessary for system verification, but it should not be used as the principal means of functional testing. This is because HIL testing is conducted at a very low abstraction level – close to the real system – and therefore combines many different effects that prevent efficient functional testing.[3]

HIL testing requires an investment in hardware that ranges from standard PCs with specialized data cards to high-end hardware racks. Fewer tests can be executed on such a system than in pure software because software testing can be more easily replicated on multiple computers. This is another reason to ensure that functionality is verified before HIL testing. If engineers find algorithmic defects during HIL testing, then the upstream verification process is probably inadequate.

Use the model and test suites as a reference design

In MBD, all key development tasks are performed at the model level. This means that any modification made to the generated code must also be made in the model. Using the model and test suites as a single source of truth throughout development promotes clear communication and efficient reuse of models and tests, not only for the current project, but also for future enhancements and derivative designs.

Put all artifacts under configuration management

Software engineers recognize the value of versioning code in a Configuration Management System (CMS). The key artifacts in MBD – models, tests, and simulation results – should also be maintained in a CMS. Managing artifacts in a CMS makes it easy for teams to rerun virtual tests and compare the current test harness with a former model state or former test vectors.

Versioning models works best when the model structure is modular rather than monolithic. A modular model structure can also accelerate development by allowing multiple engineers to work on different parts of the same system in parallel and by enabling parallelized code generation.

Perform regression tests

Software engineers use nightly builds to compile and test the most up-to-date version of the source code. This approach should be applied to modeling and simulation as well. Once an engineer defines a new test to verify a specific model behavior, that test should be integrated into the nightly build to ensure that the specific behavior still works in all subsequent modeling iterations. If the test fails at some point, then either a defect has been identified or the functionality has fundamentally changed and the test is no longer applicable.

Verify early and often

The best practices outlined in this article allow engineers to achieve early verification, lessening the time spent at the end of the development cycle testing and debugging their designs. Key to this process is MBD, which enables the use of verification as a parallel activity that occurs throughout the development process. Performing test and verification along every step of the development process means finding errors at their point of introduction. The design can be reiterated, fixed, and verified faster than in the traditional process.

References:

[1] Holzapfel, Florian, et al. Autopilotenentwicklung als Benchmark für einen durchgängigen System- und Softwareentwicklungsprozess. Garching: DGLR, 2009. Workshop: Brücke zwischen Systemdesign und Softwareentwicklung in der Luft- und Raumfahrt.

[2] Friedman, Jonathan, Prabhu, Sameer M., and Smith, Paul F. Best Practices for Establishing a Model-Based Design Culture. 2007. SAE World Congress.

[3] Schlosser, Joachim. Architektursimulation von verteilten Steuergerätesystemen. Berlin: Logos Verlag, 2006.

Guido Sandmann is the automotive marketing manager, EMEA, at MathWorks. He has more than 10 years of experience applying MathWorks products in various application areas. Guido has a degree in Computer Science from the University of Oldenburg.

Joachim Schlosser is senior team leader in the Application Engineering Group at MathWorks. He has experience as a process and methodology consultant as well as an application engineer for a MOST simulation/emulation tool. Joachim has a degree in Computer Science from the Augsburg University of Applied Sciences and a PhD in Computer Science from Technical University Munich.

Brett Murphy is technical marketing manager at MathWorks. He has extensive controls analysis, real-time software development, and systems engineering experience in the aerospace and embedded systems industries. Brett holds BS and MS degrees in Aerospace Engineering from Stanford University.

MathWorks
Linkedin: www.linkedin.com/company/the-mathworks_2
Facebook: www.facebook.com/MathWorks
Twitter: @MathWorks
www.mathworks.com

Although software verification techniques can help developers find specific types of defects in embedded C software, they can overlook some errors. To rest assured that defects aren’t slipping through the cracks, developers should apply these complementary techniques in concert.

Automated techniques such as pattern-based static code analysis, runtime memory monitoring, unit testing, and flow analysis can be used together to find bugs in an embedded C application. The following discussion will demonstrate these techniques using Parasoft C/C++test, an integrated solution for automating a broad range of best practices to improve C and C++ software development team productivity and software quality.

Sample sensor application

The recommended bug-finding strategies can be explored in the context of a simple sensor application running on an ARM  Cortex-M3 board. An application is created and uploaded to the board, but when it’s run, it doesn’t render the expected output on the LCD screen.

It’s not working, and the reason why is unclear. Debugging on the target board would be time-consuming and tedious, as debugger results would need to be analyzed manually to try to determine the real problems. Alternatively, certain tools or techniques could be applied to pinpoint errors automatically.

At this point, the two options are to debug the application with the debugger or apply an automated testing strategy to peel errors out of the code. If the application still does not work after applying the automated techniques, the debugger can be used as a last resort.

Pattern-based static code analysis

Instead of debugging, pattern-based static analysis – which is fast, easy to use, and can be applied at almost every code  change – is applied. One problem is identified by performing static analysis (see Figure 1).

This is a violation of a MISRA rule that says using assignment operators inside Boolean expressions can be risky. The intention was not to use an assignment operator, but rather a comparison operator. So this problem is fixed and the program is rerun.

There is improvement as some output is displayed on the LCD. However, the application crashes with an access violation. Once again, there is a choice to make: Use the debugger or continue applying automated error detection techniques. Given that automated error detection is very effective at finding memory corruptions such as this, performing runtime memory monitoring is the best option.

Runtime memory monitoring of the complete application

Runtime memory monitoring can be performed by applying lightweight instrumentation suitable for running on the target board. After uploading and running the instrumented application and downloading results, an error is reported (see Figure 2).

This indicates reading an array out of range at line 48. Obviously, the msgIndex variable must have had a value that was outside the bounds of the array. Going up the stack trace reveals that this print message with an out-of-range value was caused by putting an improper condition for it before calling function printMessage(). This can be fixed by relaxing the value range control inside the if statement and taking away the unnecessary condition(value <= 20).

void handleSensorValue(int value)

{

  initialize();

  int index = -1;

  if (value >= 0 && value <= 10) {

     index = VALUE_LOW;

  } else if ((value > 10) && (value <= 20)) {

     index = VALUE_HIGH;

  }

  printMessage(index, value);

}

Now, when rerunning the application, no memory errors are reported. After the application is uploaded to the board, it seems to work as expected. However, some concerns remain.

One instance of a memory overwrite was found in the code paths that were exercised, but does that mean there are no memory overwrites in the code that wasn’t exercised? Coverage analysis shows that some code has not been exercised at all. The reportSensorFailure()  function is not covered, and one branch inside the mainLoop function that calls reportSensorFailure has not been exercised at all (see again Figure 2). One way to test this code is to create a unit test (for the mainLoop function) in combination with a user stub (for the readSensor function) to simulate conditions that are difficult to reproduce during functional testing.

Unit testing with runtime memory monitoring

A test case skeleton is created and then filled with test code.  Also, a stub is added for the readSensor function to simulate a reading error. The test case is run – exercising just this one previously untested function – with runtime memory monitoring enabled. The results show that the function is now covered, but new errors are reported (see Figure 3).

The test case uncovered more memory-related errors. There is a clear problem with memory initialization (null pointers) when the failure handler is being called. Further analysis shows that an order of calls was mixed in reportSensorValue(), such that finalize() is being called before printMessage() is called, but finalize() actually frees memory used by printMessage().

void finalize()

{

  if (messages) {

     free(messages[0]);

     free(messages[1]);

     free(messages[2]);

  }

  free(messages);

}

void printMessage(int msgIndex, int value)

{

    const char* msg = messages[msgIndex];

    printf("Value: %d, State: %s\n", value, msg);

    fflush(stdout);

}

void reportSensorFailure()

{

    finalize(); 

    printMessage(ERROR_MSG, 0);

}

This order is fixed, and the test case is rerun one more time.

That resolves one of the errors reported. The next step is to address the second problem reported: AccessViolationException in the print message. This occurs because these table messages are not initialized. To resolve this, the initialize() function is called before printing the message. The repaired function looks as follows:

void reportSensorFailure()

{

  initialize();

  printMessage(ERROR, 0);

  finalize();

}

When rerunning the test, only one task is reported: an invalidated unit test case, which is not really an error. The outcome must be verified to convert this test into a regression test (see Figure 4).

Next, the entire application is run again. Coverage analysis shows that almost the entire application was covered, and the results indicate that no memory error problems occurred.

Even though the entire application was run and unit tests were created for an uncovered function, there are still some paths that are not covered. Unit test creation can continue to be employed, but it would take some time to cover all of the paths in the application. Instead, those paths can be simulated with flow analysis.

Flow analysis

Flow analysis is run to simulate different paths through the system and check if there are potential problems in those paths. Several issues are reported (see Figure 5).

There is a potential path – one that was not covered – where there can be a double free in the finalize() function. The reportSensorValue() function calls finalize(), then finalize() calls free(). Also, finalize() is called again in the mainLoop(). This can be fixed by making finalize() more intelligent:

void finalize()

{

  if (messages) {

     free(messages[0]);

     free(messages[1]);

     free(messages[2]);

     free(messages);

     messages = 0;

  }

}

Flow analysis is then run one more time. Only two problems are reported (see Figure 6).

A table with the index -1 is possibly being accessed here. This is because the integral index is set initially to -1, and there is a possible path through the if statement that does not set this integral to the correct value before calling printMessage(). Runtime analysis did not lead to this path, and this path might never be taken in real life. That is the major weakness of flow analysis compared to actual runtime memory monitoring. Flow analysis shows potential paths, not necessarily paths that will be taken during actual application execution. This potential error is fixed easily by removing the unnecessary condition (value >= 0).

void handleSensorValue(int value)

{

  initialize();

  int index = -1;

  if (value <= 10) {

     index = VALUE_LOW;

  } else {

     index = VALUE_HIGH;

  }

  printMessage(index, value);

}

The final error reported is fixed in a similar way. Now, when rerunning flow analysis, no issues are reported.

Regression testing

To ensure that everything is still working, the entire analysis is rerun. First, the application is run with runtime memory monitoring and everything seems fine. Then unit testing is run with memory monitoring and a task is reported (see Figure 7).

The unit test detected a change in the behavior of the reportSensorFailure() function. This was caused by modifications in finalize(), a change that was made to correct one of the previously reported issues. This task draws attention to the change and indicates that the test case must be reviewed. Then either the code should be corrected or the test case should be updated to show that this new behavior is actually the expected behavior. After looking at the code, it is apparent that the latter is true, and the assertion’s condition is updated.

void sensor_tests_test_reportSensorFailure()

{

    {

         messages  = 0 ;

    }

    {

        reportSensorFailure();

    CPPTEST_ASSERT(0 == ( messages ));

    }

}

As a final sanity check, the entire application is run on its own, building it in the integrated development environment without any runtime memory monitoring. The results confirm that it is working as expected.

Complementary tools

All of the testing methods applied – pattern-based static code analysis, memory analysis, unit testing, flow analysis, and regression testing – do not compete with one another, but rather complement one another. Used together, they provide an amazingly powerful tool that delivers an unparalleled level of automated error detection for embedded C software.

Marek Kucharski, president of Parasoft SA and VP of development, directs operations, sales, and development at Parasoft Corporation’s Polish subsidiary. Marek has been developing and managing software systems since he graduated from Jagiellonian University in Krakow in 1994. His professional experience includes building a wide range of software, from retail client server systems to cutting-edge development tools.

Mirosław Zielinski, C++test product development manager, is responsible for developing embedded applications of Parasoft’s C/C++test embedded testing product. Mirosław has been developing and supporting embedded systems testing frameworks since he graduated from AGH University of Science and Technology in Krakow in 2002, where his studies of automation systems and applied robotics gave him key insight into embedded software industry quality challenges.

Parasoft
626-256-3680
info@parasoft.com
Linkedin: www.linkedin.com/company/parasoft
Facebook: www.facebook.com/parasoftcorporation?ref=s
Twitter: @Parasoft
www.parasoft.com

Using traditional tools to verify that structural coverage meets safety-critical standards can be complicated since the code being tested is not the same code that will finally execute. A tool that derives precise source-level coverage metrics from execution trace data for a noninstrumented program offers a simpler alternative that supports all levels of safety certification.

Certification standards such as DO-178B for commercial avionics require evidence that the system source code is completely exercised by tests derived from requirements. Traditional tools obtain the coverage data through code instrumentation, but this complicates analysis since the code being tested is not the code that will finally execute.

A host-resident two-part technology offers an efficient and cost-effective alternative solution: a target emulator coupled with a non-intrusive coverage analyzer. The emulator is not an interpreter; instead, it dynamically translates the object code into native host instructions. As a result, test suites typically execute faster than on the actual target hardware. The coverage analyzer derives source coverage data from object branch information retrieved from program execution on the emulator, and performs any additional analysis needed for compliance with the most stringent coverage requirements.

Final verification on the target platform is simplified; it entails rerunning the tests and showing that the results are the same as on the emulator. This approach fully supports all levels of safety certification for DO-178B and for its upcoming revision, DO-178C.

Verification challenges

A major verification activity specified in safety certification standards such as DO-178B[1] is test coverage analysis, which involves demonstrating that each software requirement is met and showing that the requirements-based tests completely cover the source code. Coverage analysis raises several issues:

·    Instrumentation: A common approach is to use a tool that generates a modified (instrumented) version of the application source code, or to compile the application with a special switch to generate instrumented object code. The added code contains calls to appropriate logging functions. However, the instrumented code is not the code that will run on the final system. To use the coverage data, the developer must demonstrate that it also applies to the uninstrumented executable. This is not necessarily a simple task.

·    Target hardware: Although final software/hardware integration testing must be performed on the actual configuration to be deployed, it is both expensive and inconvenient to require the target board during component development. A host-based solution is simpler and more cost-effective.

·    Source versus object coverage: DO-178B requires coverage of the source code, but the coverage data is computed from the executing program. At the highest safety criticality (Level A), special analysis might be needed to demonstrate Modified Condition/Decision Coverage (MC/DC).

The technology described here addresses these issues. It is based on deriving source coverage metrics from execution trace data produced by a host-resident target emulator running an uninstrumented version of the application software.

DO-178B test coverage analysis

DO-178B specifies two types of test coverage analysis [1, §6.4.4]:

·    Requirements-based test coverage analysis: The developer must show traceability from each requirement to the source code that implements the requirement, and to a test suite whose execution provides confidence that the requirement is implemented correctly.

·    Structural coverage analysis: The developer must show that the code structure has been completely exercised by the requirements-based tests. If these tests do not completely cover the source code, then the developer must add further requirement(s), add further test(s), and/or remove code – referred to as “dead code” (DO-178B) or “extraneous code” (DO-178C) – that is not traceable to requirements.

The extent of the needed coverage depends on the software component’s safety criticality level. At Level C, only statement coverage is required; that is, each statement in the program must be executed at least once.

At Level B, decision coverage is required. In DO-178B parlance, a decision is a complete Boolean expression consisting of atomic Boolean terms (conditions) and Boolean operators. For example, the following Boolean expression is a decision with three conditions:

(B1 and then B2) or else B3

This example uses the Ada and then and or else short-circuit forms that evaluate their right operand only when necessary, corresponding respectively to the && and || operators in C. Decision coverage requires each decision in the program to be exercised by tests for both true and false.

At Level A, MC/DC is required:

·    Each condition in the program must be exercised by tests for both true and false.

·    Each decision in the program must be exercised by tests for both true and false.

·    Each condition must be shown to independently affect the decision’s outcome (with that condition varying while all other conditions remain fixed).

MC/DC does not require each decision to be tested with every possible combination of truth-values for its constituent conditions. This would be unrealistic for complex decisions and perhaps impossible when conditions are coupled (when the same input variable appears in multiple conditions).

Figure 1 shows a program fragment that illustrates the differences among the various kinds of structural coverage. MC/DC, which has some subtle characteristics, is discussed comprehensively in a tutorial by Hayhurst et al[2] and a detailed study by Chilenski[3].

Source versus object coverage

A commonly misunderstood requirement in DO-178B concerns the type of coverage (source versus object code) that must be demonstrated at Level A. Section 6.4.4.2 states:

The structural coverage analysis may be performed on the Source Code, unless the software level is A and the compiler generates object code that is not directly traceable to Source Code statements. Then, additional verification should be performed on the object code to establish the correctness of such generated code sequences. A compiler-generated array-bound check in the object code is an example of object code that is not directly traceable to the Source Code.

This requirement, whose wording is being revised in DO-178C, does not say that object coverage must be demonstrated for Level A. Instead, it addresses the issue of source language constructs whose compiled object code contains conditional branches or side effects not apparent from the source code. In such cases the developer must verify the generated code, for example by explaining the effect of each non-traceable object code sequence. But coverage analysis still must relate to the source code structures. Showing object code coverage is not sufficient unless further analysis can demonstrate its equivalence to source code coverage.

Target emulation through virtualization

The concept of simulating a target processor on a host system is not new, but recent advances in virtualization technology have spawned an efficient and portable approach exemplified by the open-source Quick EMUlator (QEMU) tool[4]. QEMU supports full system emulation with guest operating systems and allows simulation of specific embedded devices through machine descriptions. It runs on a host platform and, in a two-stage process, dynamically translates object code into native host instructions, using a caching scheme for efficiency. The tool first translates target code into an intermediate language and then compiles the intermediate representation into host binary instructions.

The dynamic translator operates on sections of uninstrumented target code at a time, interleaving translation (or cache fetch) with execution of the translated instructions. When QEMU starts processing a section of target code, it translates the instructions to host code until it reaches the next branch. The translated target code, known as a translated block, is stored in a cache if not already there, and its corresponding host instructions are executed. QEMU then continues translating where it left off. Because of the caching, blocks of target instructions only need to be decoded once. And in practice, because host processors are typically faster than the embedded target hardware, QEMU’s virtualization approach provides better performance than direct execution on the target.

QEMU is open-source technology that can be extended to provide additional functionality. To handle structural coverage analysis as required by DO-178B, a useful enhancement is support for generating execution traces. Two kinds of trace information are relevant:

·    Summary traces: The output identifies the address ranges of the instructions that were executed, and, for conditional branches, which branch(es) was (were) taken. The output data has bounded size (actually linear with respect to object program size), as it only reveals which instructions/branches were executed and not the entire execution history.

·    Full historical traces for specified address ranges: In addition to indicating the instructions that were executed, the output shows which branch was taken at each evaluation of the relevant conditional expressions. The size of the output data depends on the execution history.

An adapted version of QEMU that produces these execution traces is a key component of coverage analysis technology.

Coverage analysis

Although the execution trace data supplies object instruction coverage and object branch coverage information, further analysis is needed to satisfy DO-178B’s coverage objectives:

·    The traces must be mapped to source code structure, particularly to constructs in the source code (statements, decisions, conditions) that have coverage requirements.

·    The level of coverage achieved – statement, decision, MC/DC – must be assessed.

To enable this analysis, the compiler can preserve the source program’s decision structure in the object control flow graph and generate two kinds of output:

·    Debugging information (DWARF), which relates each object code instruction to a source code location (file, line, column).

·    Source Coverage Obligations (SCOs), which provide a compact representation of the program’s constructs that require evidence of fulfilling some coverage objective. SCOs capture the structure of all decisions in the program.

Using trace data from the emulator as well as the DWARF and SCO information provided by the compiler, a coverage analysis tool can deduce whether the test’s execution accomplished the desired level of coverage (statement, decision, MC/DC).

Determining whether the execution trace data implies MC/DC presents some challenges. One issue is how to infer source condition evaluation from object branch coverage. This can be handled if the program uniformly uses short-circuit forms (“and then”, “or else”) instead of the non-short-circuited operators (“and”, “or”)[5]. The compiler, as directed by an option, preserves the source code’s conditional structure in the generated object code. A second issue is whether it is possible, for efficiency reasons, to use only the summary traces and not the full historical traces. In general, the answer is “no,” and a relatively simple decision shows why:

(B1 and then B2) or else B3

The object code for this decision can be branch-covered by just three test cases, as shown in Table 1.

However,  MC/DC requires at least n+1 tests when there are n independent conditions[2], and thus four here (see again Figure 1). This means that the trace summary data (object branch coverage) is not sufficient; full historical trace data is needed. A mathematical characterization of when object branch coverage is sufficient to deduce MC/DC is given in Bordin et al[6] and Comar et al[7].

Further issues to be addressed when object code coverage is proposed as evidence for MC/DC are documented in several certification authority reports[8, Section 20][9].

Putting it all together

The target virtualization approach has been implemented as part of an effort – the Couverture (Coverage) Project[6] – to provide an open framework for coverage analysis for safety-critical software development. AdaCore’s GNATemulator tool is an adaptation of QEMU that collects the execution trace data. The GNAT compiler compiles an application source program with switches that preserve the conditional control flow in the object code and generate the DWARF and SCO data. The uninstrumented executable is then run on GNATemulator, producing the execution trace data. Using the information generated by the compiler and the emulator, the GNATcoverage tool assesses whether the required structural coverage has been achieved. If necessary, the tool analyzes the full historic trace data to verify MC/DC. Figure 2 depicts a typical development scenario.

These tools currently work on applications written in Ada, a language that is used frequently in the safety-critical domain. Future versions will support other languages, including C. Target architectures currently supported include PowerPC and LEON.

Efficient target virtualization, coupled with a tool that deduces precise source-level coverage metrics from execution trace data for a non-instrumented/unmodified user program, mark an advance in the state of the art. This technology is especially valuable in safety-critical contexts, supporting safety certification at all levels while simplifying the certification effort.

References:

[1] RTCA SC-167/EUROCAE WG-12. DO-178B – Software Considerations in Airborne Systems and Equipment Certification; December 1992.

[2] K. Hayhurst, D. Veerhusen, J, Chilenski, L. Rierson. A Practical Tutorial on Modified Condition/Decision Coverage. NASA/TM-2001-210876; May 2001.

[3] J. Chilenski. An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion. DOT/FAA/AR-01/18; April 2001.

[4] QEMU open-source processor emulator. wiki.qemu.org/Index.html

[5] G. Romanski. MCDC Coverage Analysis Using Short-Circuit Conditions; White Paper; Verocel, Inc.

[6] M. Bordin, C. Comar, T. Gingold, J. Guitton, O. Hainque, T. Quinot. “Object and Source Coverage for Critical Applications with the Couverture Open Analysis Framework.” Embedded Real-Time Software and Systems (ERTS²) 2010; Toulouse, France.

[7] C. Comar, J. Guitton, O. Hainque, T. Quinot. “Formalization and Comparison of MCDC and Object Branch Coverage Criteria.” Submitted to Embedded Real-Time Software and Systems (ERTS²) 2012.

[8] European Aviation Safety Agency. Certification Memorandum EASA CM-SWCEH-002, Issue 01; Software Aspects of Certification. August 2011.

[9] Certification Authorities Software Team (CAST). Position Paper CAST-17, Structural Coverage of Object Code (Rev 3); June 2003.

Benjamin M. Brosgol is a senior member of the technical staff at AdaCore. He has been involved with programming language design and implementation for more than 30 years, serving in the design team for Ada 95 and in expert groups for several Java Specification Requests. Ben holds a BA in Mathematics from Amherst College and MS and PhD degrees in Applied Mathematics from Harvard University.

AdaCore
212-620-7300
brosgol@adacore.com
Linkedin: www.linkedin.com/company/adacore
Facebook: www.facebook.com/pages/AdaCore/104074652961446
Twitter: @AdaCoreCompany
www.adacore.com

While Long-Term Evolution (LTE) cellular technology enables a variety of supercharged automotive infotainment applications, the novel technology introduces a number of engineering challenges regarding network handoff and antenna complexities. Designers need to evaluate these considerations before they can unlock LTE’s potential for advanced data management, cost savings, and a better user experience.

Imagine being able to take your Netflix account on your next road trip and letting your kids stream their favorite movies on demand. Imagine jumping onto a video conference for some last-minute prep for a big meeting while getting a ride to the airport. Imagine watching real-time video of upcoming traffic problems taken from other drivers’ vehicles and streamed right to your dashboard.

You don’t have to imagine much longer. High-quality video streaming to the vehicle is almost here, and new Long-Term Evolution (LTE) cellular technologies are making it happen. Network-based In-Vehicle Infotainment (IVI) systems such as Ford SYNC have been on the market for several years, offering a variety of cloud-based media, maintenance information, emergency assistance, and other services. These applications are about to get supercharged with the wide-scale deployment of LTE technology.

LTE networks operate at data rates up to 100x faster than today’s 2G and 3G cellular connections, bringing the horsepower of fixed-line home broadband connections to the vehicle. They also provide much better range, especially in rural areas that are underserved by 3G data networks.

However, LTE introduces some significant challenges for system designers and automotive engineers, including more complex network handoffs and antenna requirements and the need for ample flexibility to accommodate evolving standards and technologies. What do system designers need to know about LTE, and what can they do to make the most of it?

LTE advantages

While today’s IVI systems use an assortment of network connectivity technologies, current trends indicate that LTE is the future of networking. The Global Mobile Suppliers Association reports that, as of September 2011, 237 operators in 85 countries are deploying LTE, with 26 networks now commercially launched. In fact, LTE is the fastest developing mobile system rollout in the history of the industry, and for good reason.

First, LTE can enable a superior user experience, with increased capacity and theoretical peak download/upload speeds of 100 Mbps/50 Mbps. LTE also operates with as little as one-tenth the latency of current 3G technologies. This means that an LTE-equipped vehicle can load a standard Web page in less than a second, compared to a typical 3G network with 100 millisecond latency, which takes five seconds to load the same page regardless of connection speed. For in-car video services, this reduced latency translates to a significant and immediately noticeable improvement in the user experience.

As an IP-based technology, LTE also has a more advanced control plane and data plane system. This enables IVI systems to employ sophisticated data management and prioritization schemes and allows drivers to access multiple network services and applications simultaneously.

Unlike previous-generation cellular systems, LTE is extremely flexible and can be deployed over several different frequency bands, including frequencies currently used for 2G and 3G services. As operators refarm their spectrum for LTE, many are deploying new high-speed networks in lower spectra (especially the 1,800 MHz frequency), improving the range of cellular data services considerably when compared to 3G technologies.

Finally, LTE’s simplified IP core and transport networks, as well as its ability to reuse existing 2G cell sites for LTE services, allow operators to deploy LTE networks more quickly and inexpensively. Ultimately, this cost savings gets passed down the line, lowering the cost per bit for in-car connected services and making high-quality HD video applications a viable business proposition.

The potential of LTE for IVI services is considerable. However, LTE technology also introduces some significant engineering challenges – most notably, more complex handoffs between LTE and non-LTE networks.

Shifting between operating modes

On today’s roads, any LTE-equipped vehicle will enjoy pockets of high-speed LTE connectivity separated by long stretches with only 3G or even 2G network coverage. In this environment, having a good LTE radio is not enough. Designers need a system that can effectively navigate complicated handoffs between different technologies and maintain a consistently high-quality user experience across the network as it is today, as well as the network of the future.

In any viable IVI system, the LTE module or modem must be capable of functioning with combinations of 3G, 2G, and evolved High-Speed Packet Access (HSPA+) networks as well as LTE, potentially in multiple frequency bands depending on where the vehicle will be sold. And it’s not enough to simply hand off the session in a way that is transparent to the user. The system also must employ some intelligent decision-making capacity to choose the best possible connection at all times. If you’re videoconferencing with your team on the way to an important meeting, for example, and your car switches from LTE to 2G even though 3G service is available, it will be little consolation that you still technically have a data connection.

All of these potential modes and frequencies raise basic engineering challenges. Operating with 2G, 3G, and LTE, as well as managing handoffs from each mode to all others is a much more complex proposition. Engineers must integrate radios for each connectivity type (possibly in multiple frequencies) and test each one individually, in addition to all possible handoffs.

Given these complexities, it is essential to look for cellular solutions from vendors with broad expertise not just in LTE, but also in other 2G and 3G technologies. LTE suppliers should demonstrate their success developing solutions that operate in multimode environments.

Because LTE is being implemented in many different ways over many different frequencies, it’s also a good idea to work with a supplier who has developed a broad range of successful LTE solutions (modules, hotspots, USB modems, and so on) in different markets. Along those lines, system designers should seek global suppliers who can pursue certifications with multiple network operators worldwide and offer precertified LTE solutions for many markets.

Thinking about antennas

Antennas have been a mature, reliable technology for many years in 2G and 3G systems, but for LTE in-vehicle systems, they can be a significant challenge. LTE relies on Multiple Input Multiple Output (MIMO) antennas, which are inherently more complex than those used in 2G or 3G systems. Balanced antenna structure, coherent distance (antenna separation), polarity, and even directionality become critically important, and systems that do not properly account for these factors deliver a noticeably degraded user experience.

In addition, while the lower-frequency bands on which LTE operates improve range, they also increase electrical noise. So antennas must not only account for more complex requirements, they must do so in a noisier environment. Given the fact that many operators are rolling out LTE in existing 2G cell sites, network coverage might also be less optimal than it would be in a network built from the ground up for LTE services.

To address these factors, system designers should make sure they work with vendors who offer a high level of expertise in the specialized discipline of antenna design and testing.

Flexibility for the future

One of the biggest challenges associated with LTE technology is simply its novelty. LTE systems work – the new network deployments launching each month testify to this fact – but LTE is still very much an evolving technology. For example, while the industry is developing an IP-based voice and SMS messaging capability for LTE (the Voice-over-LTE, or VoLTE initiative), there is currently no industry-wide standard for implementing these services. In addition, LTE is evolving to LTE-Advanced, which will support even faster data rates.

When developing LTE-based infotainment systems for vehicles that will be on the road five, six, or 10 years down the line, designers need to be sure they are using solutions with enough headroom to accommodate evolving standards and technologies. They should look for programmable LTE modules and modems that will allow them to embed application and communication intelligence into the cellular module, as well as add new capabilities over time via over-the-air software updates. They should look for modules built with operating system-like processing capabilities, if not actual lightweight operating systems. Along those lines, they should seek out cellular vendors with robust development platforms that provide everything designers need to build and continually evolve in-vehicle cellular solutions. For example, Sierra Wireless offers its AirPrime AR Series of wireless modules designed specifically for the automotive industry (see Figure 1).

Don’t drive alone

The facts are clear: LTE is coming, and it will unlock a world of new possibilities for in-vehicle communications and applications. Moreover, it is just as clear that delivering LTE IVI systems is not a straightforward proposition.

With mature 2G and 3G technologies, it might have been possible to simply add a cellular modem to an otherwise isolated infotainment system. To develop a high-quality LTE solution, however, system designers will benefit greatly from working with an established supplier who can navigate the unique challenges and potential pitfalls of the technology. From shepherding the system through operator certifications to assuring the solution accounts for unique network implementations in target markets, a strong wireless technology partner can make the journey to tomorrow’s infotainment systems a much smoother ride.

Pierre Teyssier is senior VP of engineering for the M2M Embedded Solutions Business Unit at Sierra Wireless. Prior to joining Sierra Wireless, he served as VP of operations and smart business solutions at Wavecom and director of manufacturing at Axiohm, and also worked as a software engineer at Enerdis.

Sierra Wireless PTeyssier@sierrawireless.com @SierraWireless www.sierrawireless.com

Multicore Systems-on-Chip are multiplying the difficulties software developers face in enabling applications to scale linearly with available cores and fully leverage the increasing amounts of processing power available. Using Linux-based virtualization methodologies can meet the high-level requirements of multicore environments while avoiding increases in cost and complexity.

Whether software developers like it or not, and whether they’re prepared for it or not, virtually every semiconductor maker worth their salt is producing multicore Systems-on-Chip (SoCs). These SoCs typically pair two or more CPU cores with additional application-specific hardware accelerators to provide a complete system. For example, Cavium Networks, NetLogic Microsystems, and Freescale Semiconductor produce SoCs for network processing, while Texas Instruments and Broadcom make SoCs for digital media devices.

For software folks, this presents the interesting challenge of enabling applications to obtain all the processing power available from these multicore SoC environments. How can developers make sure their applications scale linearly with the available cores, as well as fully utilize the other SoC hardware components such as media accelerators and packet engines? To be clear, the scalability question is still a real science project for many applications; however, there are systems to build and products to ship, so developers can’t wait for the theoretically perfect solution.

In the past year, MontaVista Software examined numerous customer use cases in a wide range of applications including network processing, digital TV, in-vehicle infotainment, super low-power server Web hosting, and more. The goal was to understand how a Linux-based software solution could make full use of the underlying SoC hardware across a wide range of application requirements. The study identified the following high-level requirements that any solution must meet.

Multicore support

The demands of modern embedded systems are hastening the adoption of multicore SoCs. These demands are further accentuated by the requirements to run multiple systems simultaneously; thus, the solution must provide an efficient way of using and managing multicore environments.

Security

Anything downloaded to a device is insecure by definition. The solution must effectively isolate anything downloaded from the core device functions, and the downloaded applications must not be allowed to contaminate other applications.

Resource congestion

Downloaded applications must be prevented from hogging system resources. The goal is to effectively share resources such as memory, CPU time, and I/O. This sharing must allow more important system functions to have priority over less important downloaded applications.

Foreign system integration

Many environments run on top of a Linux kernel. However, these environments might require different userland libraries, as well as different kernel patches. For example, the Android system has its own device drivers and kernel patches. Ideally, the system could run any userland that runs on a Linux kernel. The kernel patches and userlands associated with these environments must be integrated with security and resource sharing in mind.

This analysis led to the development of a Linux-based architecture that maximizes the underlying power of today’s powerful multicore SoCs.

Architecture overview

To understand the overall architecture of this software, it is necessary to know a bit about modern Operating System (OS) environments, most notably virtualization technology. But be careful; there’s a lot of hype around virtualization (or, as we like to say, a lot of hype around hypervisors).

Virtualization is a method for dividing a computer’s resources into multiple execution environments. There are three major categories of virtualization in use today, with the key difference among them being the layer where virtualization occurs:

• Full virtualization and paravirtualization: These types of virtualization are used to host multiple guest OSs that are isolated from one another. While highly functional, the performance (without a great deal of optimization) is very low due to the overhead of the hypervisor and multiple OSs. Examples include QEMU, Kernel-based Virtual Machine (KVM), Zen, and VMware.
• OS resource virtualization: This type of virtualization is used to isolate and scale applications using a single OS. The advantage here is a single OS and lower overhead, typically less than 1 percent in most cases. Because there is so little overhead, the ability to scale and/or optimize performance is a huge benefit. Examples include Linux Containers and BDS Jails.
• Hardware segmentation (Asymmetric Multi-Processing or AMP): This  high-performance configuration dedicates hardware to specific applications running in user mode for maximum performance. This can be achieved using a simple runtime executive or leveraging OS resource virtualization and processor core affinity capability to dedicate cores and I/O to processes with almost no overhead.

These types of virtualization offer different performance characteristics, require different setup and maintenance overhead, introduce unique levels of complexity into the runtime environment, and address different problems.

While the industry is currently focused on pushing fully virtualized hypervisors as the one-size-fits-all solution to multicore optimization, the reality is that embedded developers need a range of options that can be tailored to specific application needs. Developers will require some combination of one or more of these virtualization technologies to deliver products that fit within hardware constraints and meet design performance characteristics. In short, the trick is to match the application with the right OS services to meet the overall system requirements, which can include performance, reliability, and security.

MontaVista provides three methods of virtualization based on nonproprietary, open-source Linux technology and supported across multiple processor architectures. Because it is a single runtime, there is one compiler and one set of tools that can be used for any use case or combination of use cases. Figure 1 shows an overall picture of this approach. These three methods are:

• KVM Hypervisor (full virtualization)
• Linux Containers (OS resource virtualization)
• MontaVista Bare Metal Engine (OS resource virtualization and SoC hardware segmentation)

Microserver use case

The idea behind microservers is to utilize smaller, more energy-efficient processors to lower the physical and energy consumption footprint of a class of Web-centric IT applications. For certain workloads, several low-power processors can be more efficient than fewer, more powerful processors. Cavium Octeon processors and those from other semiconductor suppliers are well-suited to meet the density and power efficiency requirements underlying the microserver concept on the basis of the power efficiency of the cores themselves. These SoCs also include dedicated hardware to handle the front-end security and encryption/decryption processing that Web-based applications require.

From a software perspective, MontaVista Linux Containers and Bare Metal Engine technology help complete the picture. Containers are used to provide OS-level virtualization, allowing very efficient virtualization of the workload requirements. For example, Containers can be used to host thousands of independent websites, each securely isolated from each other. Containers allow the precise control of runtime resources allocated to each container, so each website can be limited to the performance levels the customer has purchased. Or, more importantly, a rogue website can be stopped from over consuming resources using the same mechanisms, thus thwarting a denial-of-service type of attack.

Bare Metal Engine provides the runtime environment for the security and encryption/decryption operations each of these hosted websites requires. For example, a 32-core SoC can utilize most of the cores for application processing with a few dedicated to packet processing, all controlled by one Linux instance.

Linux offers a simple solution

It is a widely held misconception that a combination of Linux and either a Real-Time Operating System (RTOS) or simple runtime environment must be utilized to fully realize the high performance available with multicore processors. Fueling this misconception is the thought that Linux itself is incapable of meeting the requirements because it is too big, too slow, and not real-time. This fallacy also drives the requirement that hypervisors and/or virtualization must mediate and isolate the different runtime environments and facilitate intercommunication among them. Often it is the RTOS vendors themselves who perpetuate this erroneous belief.

In the end, these misconceptions about Linux drive added complexity and costs into the development process. Complexity increases due to multiple runtime and development environments (one each for Linux, the RTOS, and possibly the hypervisor). Costs increase because of royalties for the proprietary RTOS and hypervisor, not to mention the added costs created by the development complexity itself, with more developers needed for a longer period of time.

The approach to use Linux everywhere and fix it where it might not meet some requirements results in a single OS environment, single tool chain, and common development and debugging tools for all aspects of the application. As Einstein said, “Make everything as simple as possible, but not simpler.”

Jim Ready is the CTO of MontaVista Software and a recognized authority in the embedded systems and real-time software industry. The cofounder of Ready Systems, he developed the first commercially viable RTOS product: the VRTX real-time kernel. Jim invented the category of embedded Linux commercialization in 1999 when he founded MontaVista Software to provide the Linux OS to the embedded systems market and embedded system expertise to the open-source Linux community.

Patrick MacCartee is a director of product management at MontaVista Software in charge of hardware enablement, pricing, and channel strategies. Patrick has worked in high tech for more than 10 years at Intel and MontaVista Software. He is also responsible for managing the MontaVista Linux 6 and Carrier Grade Edition products.

MontaVista Software marketing@mvista.com @mvista www.mvista.com

Model-Based Design enables rapid prototyping, easy debugging, accurate analysis, and a host of other system design benefits. Ken outlines the fundamentals of Model-Based Design and shows how techniques and tools such as automatic code generation and parallel computing with multicore processors can enhance simulation.

ECD: What trends do you see in embedded design?

KARNOFSKY: The overarching trend is accelerating integration of technologies at multiple levels, from hardware to applications. Interaction between system components is causing engineering disciplines to collaborate more effectively. One example stems from the growth of hardware/software coprocessing. Highly integrated computing architectures with multiple cores and hardware accelerators have performance and cost advantages, but they’re difficult to design and debug using conventional tools and processes.

A similar phenomenon is happening at the interface to the real world, where analog meets digital. We’re seeing increasing interest in mixed-signal system design, helping engineers use digital algorithms to lower cost and power consumption and improve the performance of sensors, RF transceivers, and other analog electronics.

Another integration trend is the increasing connectivity of everything. Smart everything – phones, cars, homes, and the power grid contain embedded devices that connect to the Internet and generate huge amounts of data. This requires additional expertise to specify and design the algorithms to communicate and process this data.

Furthermore, the proliferation of inexpensive hardware is transforming the experience of engineering students. Devices like Arduino and BeagleBoard make the integration of classroom and project-based learning a reality. We’re seeing a lot of curriculum innovation that involves integrating MATLAB and Simulink with these hardware platforms. These innovations will make today’s students better prepared for the work they’ll do in industry.

ECD: What are the advantages of Model-Based Design?

KARNOFSKY: Model-Based Design (MBD) uses a system model as an executable specification throughout development (see example in Figure 1). The model represents the embedded system and its environment and enables:

• System-level design and simulation. The system model can include every element that affects system behavior – algorithms, logic, physical components, and IP. Simulation enables analysis of system performance in conditions otherwise too expensive, risky, or time-consuming to consider.
• Automatic code generation. Generate C, C++, or HDL code from models for rapid prototyping and production software and hardware implementation. The generated code can be optimized and combined with handwritten code.
• Continuous test and verification. The system model captures requirements in an executable specification. It also provides a reusable test harness for virtual integration and hardware-in-the-loop testing.

In 2010, Embedded Market Forecasters conducted an analysis to determine the impact of MBD on the Total Cost of Development (TCD). Data was collected from companies worldwide in the automotive, aerospace, communications, industrial automation, and medical industries. The results show that:

• Development teams that used MBD had an average 37 percent lower TCD compared to teams that did not use MBD.
• The number of developers used per project was smaller for MBD across all geographic areas and all vertical markets examined.
• The percent of developer months lost to design cancellation or projects behind schedule was consistently less for MBD development projects across all verticals and geographic areas.

ECD: Can embedded design simulation replace physical prototypes?

KARNOFSKY: Simulation with system models enables rigorous system design verification with less time and expense than physical prototypes. This can significantly reduce the number of prototypes required, and in some cases eliminate them altogether.

The primary advantages of simulation are speed of design iterations and early visibility into system behavior for debugging, analysis, optimization, and verification. This reduces the design cost by identifying errors earlier in the development process, often without physical prototypes.

Engineers working toward an optimized design must develop their software and physical system together. Today, however, advances in tools and techniques enable highly accurate physical models that can replace physical prototypes of electrical, mechanical, and other components.

Many engineering teams also rely on automatic code generation with off-the-shelf rapid prototyping hardware (see Figure 2). Code generation enables reuse of algorithm and physical system models for real-time simulation and hardware-in-the-loop testing. These systems help engineers test more thoroughly by starting before hardware is available and simulating conditions that would be dangerous or costly to examine with the real system. They also significantly reduce development time and cost in the testing lab.

ECD: How does the latest wave of multicore processors affect design analysis and optimization?

KARNOFSKY: Multicore processors affect design in two ways. First, multicore PCs and servers provide an opportunity for dramatic reductions in simulation time. Parallel computing tools that are integrated with an MBD environment can take advantage of this computing power to accelerate execution of large-scale problems such as Monte Carlo simulations. For more information on how parallel computing techniques can speed up parameter optimization for a complex control system, see the MATLAB Digest article “Improving Simulink Design Optimization Performance Using Parallel Computing” at www.mathworks.com.

Second, multicore embedded processors make software design and development much more complex. Developing applications for these devices requires an understanding of the target processing architecture and careful assessment of execution performance, latency, and task dependencies. This is driving the need for tools that support the mapping of computationally intensive algorithms onto multicore architectures, as well as simulating execution in the target environment.

Ken Karnofsky is the senior strategist for signal processing applications at MathWorks. Through his 20 years of experience, first with BBN Technologies, then with MathWorks, Ken has been involved in development and marketing of software for signal processing and data analysis technologies. Ken holds a degree in Systems Engineering from the University of Pennsylvania.

MathWorks 508-647-7443 www.mathworks.com

Just like building a house that can withstand storms, building a smart grid device requires a secure operating system as the foundation. Starting with a proven, secure operating system deployed in thousands of critical applications forms the groundwork on which security for an intelligent grid can be built.

The smart energy market is hot. Just like a pressure cooker, there are many opposing forces pushing and pulling on the smart grid infrastructure, creating friction, heat, and opportunity. At the core, business challenges are colliding with technical and political challenges. There is money to be made by the utilities as well as grid component suppliers, and consumers will benefit from lower energy costs and more reliable energy. Politically, opposing forces are spouting concerns with privacy of information and even health hazards caused by the deployment and operation of smart energy equipment.

In its current state, the smart grid infrastructure is antiquated, completely insecure, incapable of supporting future energy demands, and not resilient to attack or system-level failures. That all said, utilities are rushing technology to market to provide automation and efficiency without any real consideration for security or reliability in their devices. At the next level down, in homes and businesses, the same can be said of the smart appliances that are being deployed.

Who is factoring security and reliability into their devices? There are huge ramifications for massively deploying insecure technology. Unfortunately, given the business drivers, it will likely take a catastrophic event to trigger government, utilities, and component manufacturers to come to the realization that they cannot sacrifice security and reliability without some different technology coming to bear.

Why security is so important

Much has been written on the security concerns of the grid and its systems, particularly Supervisory Control and Data Acquisition (SCADA, referring to computer systems that monitor and control industrial-, infrastructure-, or facility-based processes). Many in the SCADA world have heard of the Stuxnet worm, which was a deliberate cyber attack on a particular industrial system. In the future, smart people with misguided initiatives and time on their hands will start tapping into the devices in the home and take control of them, creating havoc, disrupting energy delivery, and stealing valuable private information. These are very real concerns for utilities and consumers, with enormous and far-reaching consequences.

Fortunately, there is hope for the smart grid as a whole. It will take time, but with an appropriate focus on protecting high-valued assets, smart grid component suppliers can leverage proven and certified technologies deployed in the aerospace, defense, industrial, and medical markets to make the grid more resilient and protected against component- and system-level failures as well as deliberate attacks.

For the grid to succeed, the components that make up the grid and the information and energy that traverse the grid must be secure and resilient. These elements must be secured, protected, and authenticated. Furthermore, when breakdowns occur, the grid must be able to self-heal and rapidly respond and recover from system failures. For system and device architects alike, the concept of defense in depth can and should be applied to people, operations, and the network of intelligent devices that comprise the grid. Single points of access and control are targets for attack and system failure. Layers of protection and separation of criticality need to exist to protect high-valued assets such as the systems monitoring and controlling the power grid.

Leveraging expertise with a certified operating system

To enable these capabilities in a cost-effective manner, smart grid device manufacturers should look to proven technologies and companies with experience deploying secure embedded devices in domains and applications like aerospace and defense that demand ultimate security and reliability. In these domains people’s lives are at stake, and systems must be designed to be absolutely secure, safe, and reliable.

The Green Hills Platform for Smart Energy provides the INTEGRITY certified Operating System (OS) to protect high-valued assets from well-funded attackers. INTEGRITY is the first and only OS technology to date to achieve certification from the National Security Agency for Evaluation Assurance Level (EAL) 6+ High Robustness, the highest level of security achieved for any commercial software. Green Hills, in business for nearly 30 years, has proven expertise in architecting secure, safe, and reliable embedded solutions. To address the software development of these complex and vital systems, developers utilize the best-in-class MULTI integrated development environment from Green Hills, which expedites the development, debugging, testing, and deployment of high-assurance real-time embedded applications.

Before looking at where the technology could be deployed, it is important to understand how and where this technology has been used. At the heart of the Green Hills platform for smart energy is an industry-proven real-time separation kernel, deployed in numerous safety- and security-critical applications such as the Joint Strike Fighter and the Joint Tactical Radio System.

The separation kernel uses hardware memory protection to create partitions and isolate and protect critical embedded applications. This ensures that each partition has the resources it needs to run correctly while also protecting the application from malicious attacks or application failures in other partitions. For complex security systems, applications with differing levels of security requirements can run in their own partitions. This architecture provides the necessary security and reliability capabilities compared to legacy applications running in a single address space.

Device-level decisions

When grid component suppliers design their devices, they make significant trade-offs between hardware cost, field upgradeability, power requirements, performance requirements, and hopefully security requirements. Manufacturers of intelligent devices need to determine what information and which devices need to be protected and how they will protect them.

With the INTEGRITY separation kernel, intrusion prevention and detection, health monitoring, firewalls, encryption, data protection, event logging, and communication stacks are all examples of applications naturally separated into partitions, enabling the device to be more resilient and secure. For in-transit data protection, support for IPv4/IPv6 with protocols such as SSH, SSL, IPsec, IKE, and RADIUS is possible. In the wireless world, WPA and WPA2 provide the latest in wireless security. INTEGRITY intrinsically supports Multiple Independent Levels of Security (MILS) with separation, data isolation, and information flow control.

Virtualization adds to resilience

Virtualization enables disparate systems to be consolidated onto a single platform, allowing more functionality to reside on a single device and thus reducing bill of material cost. Example applications include network devices or utility applications that have applications running across mixed OSs.

A concrete example is a home energy console with a Graphical User Interface (GUI) running Android that collects information from appliances in the home and transfers confidential household or business information out to a concentrator with a trusted real-time application. In this example, Green Hills Secure Virtualization enables the GUI to run in its own secure virtual machine while the communications run in an independent virtual machine. The separation kernel protects the overall system from being affected by insecure applications running in isolated partitions.

Separation and virtualization make the system more resilient by giving the developer the ability to separate critical from non-critical tasks. In this way, the system can adapt to problems, self-heal, restart applications, and safely and securely continue to operate under adverse conditions. Figure 1 depicts an example architecture utilizing separation and virtualization.

For the smart grid, there are many areas where certified separation kernel technology can make the overall system more resilient to attack. With the large installed base of insecure legacy devices in the grid, a separation kernel with virtualization enables entirely insecure devices to safely and securely operate independently within a partition. This allows the utilities and device manufacturers to leverage and reuse legacy devices in the overall grid architecture by isolating insecure partitions from security-critical components.

From the utility perspective, the ability to enable components in the grid with secure partitions for trusted communication and health monitoring means utilities can remotely monitor, control, and upgrade fielded components as required and as policy permits. This will enable both automated and manual response mechanisms to grid performance and functionality issues.

Getting the foundation right

Several other topics must be addressed to improve the grid’s resiliency, such as device and information authentication, secure boot, secure wireless and wired protocols, government policies, physical access control, and operational management. Those items cannot be properly addressed on top of an insecure OS.

At the heart of the grid are devices that collect, transmit, receive, and process valuable private information. The foundation for the success of the grid depends on these devices being properly secured and protected. Therefore, device manufacturers should leverage companies and technology that have a proven track record for delivering certified security and safety solutions in critical environments.

Jim McElroy is director of industry business development for Green Hills Software, where he is responsible for defining vertical solutions for clients and expanding business in sectors such as medical, smart energy, and transportation. He has worked in the embedded industry for more than 20 years, maintaining management and engineering positions at Esterel Technologies, Telelogic North America, I-Logix, Raytheon, and Lockheed Martin. Jim holds a BS in Computer Science from the University of Massachusetts and an MS in Computer Science from Fitchburg State College.

Green Hills Software 805-965-6044 www.ghs.com

Due to the rise of multicore processors, programmers increasingly need to write multithreaded code. Pitfalls such as race conditions, starvation, and deadlock make it hard to get such code right, and traditional testing can only help so much. New approaches based on static analysis are proving effective at quickly finding difficult bugs.

Although decades of advances in miniaturization have yielded enormous performance gains for single processors, it appears this era is coming to a close. The best bet for achieving significant additional performance with single chips is through multiple cores, but only if software can be programmed to take advantage of them.

Unfortunately, concurrent programming is difficult. Even expert-level programmers familiar only with single-threaded programming often fail to appreciate that concurrent programs are susceptible to entirely new classes of defects such as race conditions, deadlocks, and starvation. It is difficult for humans to reason about concurrent programs, and some aspects of programming languages themselves are ill-suited to concurrency. Consequently, experts frequently stumble over these hazards. The following discussion describes common concurrency pitfalls and explains how static analysis tools can find such defects without executing the program.

Consequences of race conditions

A race condition arises when multiple threads of execution access a shared piece of data and at least one of them changes the value of that data without an explicit synchronization operation to separate the accesses. Depending on the interleaving of the two threads, the system can be left in an inconsistent state.

Race conditions are especially insidious because they can lurk undetected indefinitely, and only show up in rare circumstances exhibiting mysterious symptoms that are difficult to diagnose and reproduce. In particular, they are likely to survive through testing into deployed software. At best, this means increased development times; at worst, the consequences can be devastating.

One reason the Northeast Blackout of 2003 was so widespread was that a race condition in a computerized energy management system caused misleading information to be communicated to the operators. As Kevin Poulsen noted in a 2004 article (www.securityfocus.com/news/8412), “the bug had a window of opportunity measured in milliseconds.” The chances of a problem like this manifesting during testing are infinitesimal. In another case, a race condition in iOS 4.0 through 4.1 (now fixed) meant that any person with physical access to an iPhone 3G or later could bypass its passcode lock under certain conditions.

An example of a simple race condition is shown in Figure 1. A manufacturing assembly line with entry and exit sensors maintains a running count of the items currently on the line. This count is incremented every time an item enters the line and decremented every time an item reaches the end of the line and exits. If an item enters the line at the same time that another item exits, the count should be incremented and then decremented (or vice versa) for a net change of zero. However, normal increment and decrement are not atomic operations; they are composed of a sequence of individual instructions that first loads the value from memory, then modifies it locally, and finally stores it back in memory. If the updating transactions are processed in a multithreaded system without sufficient safeguards, a race condition can arise because the sensors read and write a shared piece of data: the count. The interleaving in Figure 1 results in an incorrect count of 69. There are also interleavings that can result in an incorrect count of 71, as well as some that correctly result in a count of 70.

For this example and for race condition bugs in general, standard debugging techniques may be ineffective for several reasons.

Rare occurrence means a reduced chance of noticing there is a problem. If the problem manifests infrequently, it may never show up during testing. The issue is twofold. Firstly, the number of possible interleavings of the instructions in two threads can be huge and increases enormously as the number of instructions grows. This phenomenon is known as combinatorial explosion. If thread A executes M instructions and thread B executes N;instructions, the possible interleavings of the two threads are:

For example, given two trivial threads with 10 instructions each, there are 184,756 possible interleavings of the instructions. Real-world software is large and complex; testing every interleaving is simply impossible. Secondly, even if testers can identify a few interleavings that merit inspection, it is difficult to set up test cases to ensure they actually occur because thread scheduling can be highly nondeterministic.

If exhaustive testing is intractable, then what can developers do? One extremely useful approach is static analysis. Advanced static analysis tools such as CodeSonar use highly sophisticated symbolic execution techniques to consider many possible execution paths and interleavings at once. These techniques can find race conditions and other concurrency errors without needing to execute the program.

Several factors make race condition diagnosis difficult. Firstly, the symptoms can be perplexing. In the Figure 1 example, the running count will usually be correct, but sometimes too high and other times too low. Secondly, programmers unaccustomed to considering the particular pitfalls of multithreaded programming may spend a lot of time puzzling over the code before the possibility of a race condition occurs to them. Advanced static analysis tools are especially helpful in this regard. They identify race conditions by examining patterns of access to shared memory locations; that is, they focus on the race itself, not its symptoms. When a race condition is identified, an advanced static analysis tool will report it along with supporting information to aid the user in evaluation and debugging. The onus on the programmer is substantially reduced.

More complexity, more bugs

Race conditions are typically avoided by using locks to protect shared resources. However, locks can introduce performance bottlenecks that might prevent the program from taking advantage of the full potential of multiple cores, so programmers must exercise care in using them. It can be tricky to write code that uses locks effectively, and this complexity can lead to a different set of problems, namely deadlock and starvation.

In a deadlock, two or more threads prevent each other from making progress because each holds a lock needed by another. Figure 2 shows how a deadlock can arise with two locks used to protect two shared variables. In this example, multiple assembly lines share a count of the total number of items currently under assembly, and a second bad_items value records how many finished items failed quality control. One thread acquires the lock on count, another acquires the lock on bad_items. Neither thread can obtain the second lock it needs; thus neither can carry out its operations, nor can it get to the point where it will release its lock. As neither update can be completed, both threads are completely stuck.

Static analysis tools can identify software at risk of deadlock by flagging situations where the same locks can be acquired in different orders by different threads, such as the threads shown in Figure 2. Eliminating all such cases is sufficient to ensure that the system cannot become deadlocked.

Starvation is another problem that occurs in multithreaded programs that use locks. A thread can starve if it is waiting for a resource currently held by another thread that takes a very long time. For example, suppose the aforementioned manufacturing automation system includes a regular audit thread that examines all entry and exit records to ensure that the running count matches total items entering less total items exiting. The audit thread would need to hold locks on the count and on all sensors, so all updates must wait for the audit to finish. If the audit runs for a long time, updates can be significantly delayed. If it runs too long, the next audit may manage to acquire all the locks and start running before the outstanding thread can make any progress. In the worst case, some or all of the updates may never have the opportunity to run.

Static analysis can provide significant value here by posing questions such as, “Is there a call to a long-running library function while a lock is held?” Tools such as CodeSonar also provide mechanisms for users to add their own checks. If an in-house function f() is known to have a long running time, engineers could add a custom check that triggers a warning whenever f() is called by a thread that holds one or more locks.

Multithreading adds entirely new classes of potential bugs to those that embedded developers must consider, making it significantly more difficult to find bugs of all kinds. The latest generation of static analysis tools can help with both of these issues.

Paul Anderson is VP of engineering at GrammaTech, where he manages the engineering team and architects static analysis tools. He has worked in the software industry for 20 years, helping organizations including NASA, the FDA, the FAA, MITRE, Draper Laboratory, GE, Lockheed Martin, and Boeing apply automated code analysis to critical projects. He received his BSc from King’s College London and his PhD in Computer Science from City University London.

GrammaTech paul@grammatech.com www.grammatech.com

You’ve been diligent and used static code analysis to identify defects early during development. Great. So now what? Finding defects is just the first step in the process of ensuring software integrity. Contextual information on every identified defect is essential to prioritize fixes and maintain bug-free software.

Embedded software is ubiquitous and provides critical functionality in a variety of devices, from the latest smartphones and gaming gadgets to life-saving medical devices. Engineering organizations creating embedded software understand that ensuring quality of code is a key differentiator and competitive advantage. Along with other methods of testing and verification, many companies have taken advantage of the benefits of code testing with modern static analysis to identify defects early in development. During the past few years, various reports by embedded market research firm VDC Research indicate strong growth in companies adopting static analysis as a critical test automation tool. Modern static analysis is arguably the most cost-effective, automated, and repeatable way to meet the challenge of ensuring the quality of complex software.

A strong reason driving this growth is that the technology used to identify critical defects such as memory corruptions, resource leaks, null pointer dereferences, and invalid memory accesses has matured to a point where finding large numbers of hard-to-find defects that traverse function and file boundaries can now be accomplished accurately, resulting in a very small number of false positives. However, the real innovation lies in providing contextual information for every defect identified. A developer needs to know why the defect exists, what impact it will have, and where it needs to be fixed.

The answer to the question of where it needs to be fixed is not as simple as knowing the file name and line number. Code branching and merging for versioning, reuse of code, and reuse of code components for development productivity allow a defect to make its way into multiple versions and products.

Consider the case of a software team with multiple branches for various versions of the product. A bug in one of these branches might exist in one or more other branches due to code replication. In another case, consider a team creating the framework to support applications for smartphones. Because they might be porting the framework onto various platforms like Windows, Android, or iPhone, it is critical that the static analysis results clearly indicate whether identified defects exist in just one place or on multiple platforms. Similarly, when software is created by aggregating from multiple sources, it is a nightmare when a particular component is used in various products, as a defect in one third-party component might end up affecting all the different products that include it.

Multiple branches for different versions of an operating system

Imagine a software development team responsible for creating a new Operating System (OS) for mobile smartphones. Because multiple mobile phone vendors (OEMs) must be supported, every vendor in the source control management system needs a development branch. In addition, each vendor typically has multiple branches for different releases and product generations. The picture starts to get complex very quickly.

Static analysis performed on every branch of the code produces a list of defects. However, depending on when a defect is introduced, it could exist in all versions or a subset. When looking at a single defect in isolation in a single branch, the challenge for developers is that they can’t gauge the severity of the defect without knowing where else it is present. A defect that is not limited to a single version or one OEM client would be severe, and fixing it would need to be prioritized over anything else. Additionally, a developer writing code to fix the defect needs to know exactly which branches in the source control management system the fix needs to be checked in. Analysis results that pinpoint the exact location where the defect exists and provide information such as the branches where defects occur are highly valuable to developers (see Figure 1).

A single framework for multiple platforms

On the flip side of branching, there is often a need to write code designed to run on multiple platforms. A software component such as a framework for mobile applications is usually built to run on various types of mobile phone platforms. For embedded devices, a common requirement is to build 32- and 64-bit versions of the same code base. Let’s take a simple example:

gcc –m32 -c foo.c

// 32-bit compile. Contains a null pointer dereference defect.

gcc -c foo.c

// 64-bit compile. Contains the same null pointer dereference defect.

A defect in foo.c that gets triggered in both 32- and 64-bit binaries will be detected and reported as a single defect. However, because the source code is the same, a sophisticated analysis would not report it as a duplicate defect. Duplicates are as harmful as false positives in losing the developer’s trust in the static analysis solution.

Sharing common code components

In this final example, consider a team developing the platform software for a family of networking switches. Because the functionality provided by the platform software must be implemented in all products, this code component will be shared (see Figure 2). For developers working on this team, the best assessment of the severity of a defect reported by static analysis is not only the impact it will have on one switch product, but also information on all the products that use this platform software component.

A product is usually created by combining many such shared components. Each component is not only a project itself, but also a part of various other projects using it. The analysis result needs to identify that a defect in this shared component has an impact on the various projects using it.

Taking the guesswork out of code testing

The adoption of modern developer-side testing methods such as static analysis is a positive trend in the embedded software industry. The technology has matured to a point where it is a strong weapon in the software engineer’s arsenal. Without needing to create elaborate test cases and testing infrastructures, static analysis automatically finds critical defects as code is written and compiled. However, for static analysis to become a developer’s most valuable tool, the analysis must provide answers to questions such as “What is the impact of this defect?” and “Where do I need to check in the fix?” to help prioritize fixing the identified defects and take the guesswork out of ensuring that software is as bug-free as possible.

Rutul Dave is senior development manager at Coverity. He has several years of software development experience in embedded and real-time systems, including work developing bleeding-edge technology systems at Procket Networks, Topspin Communications, and Cisco Systems. When not evangelizing about the benefits of software integrity, Rutul scratches the coding itch by developing mobile apps and understanding the Linux kernel. He received his Master’s in Computer Science with a focus on networking and communications systems from the University of Southern California.

Coverity info@coverity.com www.coverity.com

As embedded devices grow in complexity and incorporate more and more features already available in the PC world, it may be time to take another look at the Windows Embedded software suite.

Embedded devices are beginning to look a lot like desktop computers, as customers demand graphical interfaces, high-speed communications, multimedia, and full integration with Internet-delivered data and services. With these features already built into the Windows operating system, it is only natural for developers to consider variations of the Windows operating system for new embedded products. Microsoft’s latest Windows Embedded portfolio of platforms and technologies offers the developer plenty of features and functionality to support the next generation of highly complex embedded devices.

Software complexity is just one of several reasons to consider a Windows variation for embedded applications. Other advantages of a Windows-based operating system are the availability of a large number of skilled programmers, familiar development tools, and extensive third-party hardware and software support. Newer embedded devices also require local data security for remote applications and network security software to safely connect with remote databases and services. Windows Embedded Standard 7 includes several new security features, including BitLocker and DirectAccess to deal with these cases. For local security, BitLocker encrypts data stored on protected volumes, so if a hard drive is removed from the system, the data is unreadable. For communications security, DirectAccess automatically creates a secure connection between client systems and the company server without the need to initiate a Virtual Private Network (VPN) session. Microsoft also spends considerable time and resources testing and fixing security flaws for both desktop and embedded products.

Thousands of image components

Code size is often listed as a disadvantage for Windows-based designs. However, Windows Embedded Standard 7 is broken down into thousands of components. It’s possible to pick from predesigned templates or create a completely unique software configuration, depending on your application. You can use Microsoft tools such as Image Builder Wizard to select from a few options or Image Configuration Editor to add, remove, or configure any functionality in Standard 7 automatically. You can create a specialized embedded operating system image that is as small as 600 MB. (The standard Windows image is about 16 GB.) After the image is running, you can make additional changes manually. Such changes could include installing and configuring software and drivers or customizing the Windows Welcome Screen.

Another frequently cited reason against developing a Windows-based embedded product is the poor response to real-time inputs. Windows Embedded Standard 7 has no inherent real-time capabilities. Thread switch times can be excessive, depending on software activity. For these applications, Microsoft recommends Windows Embedded Compact 7, an updated version of Windows Embedded CE, or third-party, real-time plug-ins that can be used to support a wide range of real-time, small-footprint enterprise and consumer devices. TenAsys is one of several vendors who offer real-time virtualization software compatible with the Windows Embedded Standard 7 platform. The TenAsys INtime RTOS allows you to combine the high-level features of Windows with a real-time, deterministic operating system.

Embedded off-the-shelf modules

One of the fastest methods to develop a new embedded device is to combine a pre-engineered, off-the-shelf module with a compatible and tested operating system. For example, Advantech supports Windows Embedded Standard 7 for a number of embedded boards, including the SOM-5890 COM Express module (Figure 1), which also fits digital signage applications.

The Advantech 4.92-inch x 3.74-inch SOM-5890 COM Express module is compliant with the newly released PICMG COM.0 R2.0 Type 6 specification and offers HDMI, DVI, and DisplayPort video interfaces as well as SVDO, LVDS, and VGA output. The COM Express module is based on the Intel Core i7 processor and Intel QM67 Express chipset and supports graphic intensive, multi-display applications. The SOM-5890 supports up to 16 GB of dual-channel DDR3 memory and extensive interface expansion for up to three DDIs, multiple PCI Express lanes, USB 2.0 ports, and a Gigabit Ethernet interface along with serial and general-purpose I/O ports. The SOM-5890 board includes specialized support for Windows Embedded Standard 7, and the module ships with Advantech’s iManager software and related APIs. (*See below for acronyms.)

Digital signs capture video analytics

One application that vividly demonstrates the ability of Windows Embedded Standard 7 to handle complex requirements is the Next-Generation Digital Signage project cosponsored by Microsoft, Intel, and NEC. Digital signage has become ubiquitous in the retail, transportation, education, health care, and lodging markets, delivering high-speed information and advertising content to a wide range of consumers. The Next-Generation Digital Signage project presents advertisers with new opportunities through the use of anonymous video analytics. As consumers pass by and look at the system screen, a built-in camera captures images. System software stores data such as gender, age, length of visit, and time of day to allow advertisers to tailor their content and graphics based on expected demographics. The system can also present daily specials, downloadable coupons, store maps, and other information in real time to respond to customer gestures, motion, or touch-screen inquiries (Figure 2).

In addition to Microsoft’s Windows Embedded Standard 7 operating system, the Next-Generation Digital Signage project relies on a group of technologies now available with the 2nd generation Intel Core architecture. This new architecture includes numerous graphics enhancements such as the integral graphics processor for high-definition hardware image decoding, the Intel Advanced Vector Extensions (AVX) instruction set for faster floating-point capabilities, and a modular design to simplify upgrades. The graphics section comprises an array of parallel execution units for 3D applications and hardware acceleration for high-speed encoding/decoding of high-definition video. These new multicore processors also utilize the Intel Active Management Technology (AMT) and Intel vPro Technology to enable remote system management for monitoring, troubleshooting, and content updates even when powered down.

Standards reduce fragmentation

NEC is creating the standardized hardware platform for the Next-Generation Digital Signage project, which combines a specialized display system that combines interactivity, audience measurement, and remote management. The NEC controller module and display are based on Intel’s Open Pluggable Specification (OPS) defining the electrical, mechanical, and thermal specifications for a plug-in module containing the computing system necessary to drive a digital signage display panel. Intel created the specification to reduce fragmentation within the digital signage market and to simplify installation, usage, maintenance, and upgrades. The OPS makes it possible for digital signage manufacturers to rapidly deploy large numbers of interoperable systems while reducing development, implementation, and support costs.

In addition to the Windows Embedded Standard operating system formerly known as XPe, Microsoft has defined or renamed several specialized versions of Windows Embedded including Compact (formerly CE), POSReady (WEPOS), Enterprise, Automotive, Server, Thin Client, and Handheld. Unique documentation, examples, and possible hardware selections to fit the category support each of these variants. Microsoft also recently released the Windows Embedded Device Manager, which allows you to deploy and update images for all your embedded devices from a single tool. With all these platforms and support tools, developers have plenty of options to handle the escalating complexity of tomorrow’s embedded projects.

Warren Webb’s background includes more than 30 years as an engineer and entrepreneur developing high-tech products for the aerospace and health care industries. Most recently, he has been writing articles on hardware design, software development, and emerging technologies for international trade magazines. Warren holds an MBA from Pepperdine University, an MS in Electrical Engineering from San Diego State, and a High Honors BSEE from the University of Tennessee.

More online:

Migrating to Windows Embedded Standard 7 white paper http://opsy.st/hxzypt

Intel Core Processor Family webcast http://opsy.st/gZwBIy

OEMs and integrators can build cost-effective “superphones” – like the “ObamaBerry” or Jack Bauer’s phone on the Fox TV show “24” – but for real military personnel and members of the intelligence community. To do this, a key consideration is understanding architectural options for melding standard commercial smartphone device hardware with mobile virtualization and open source software.

Military personnel and national security operatives are probably the most mobile workers in public service today, and depend on specialized and secure communications equipment to fulfill their missions at the office and in the field.

In the past, such secure communications devices – digital field radios, police/fire radio transceivers, and so on – emerged from highly proprietary design and acquisition cycles, resulting in systems that were hard to build, expensive to acquire, difficult to maintain, and impossible to upgrade. And they were yet another device for personnel to carry in addition to handsets and notebook computers.

Today, in lockstep with initiatives previously launched by the U.S. government, federal agency IT management and mobile device integrators and contractors are leaving behind the world of proprietary legacy systems and looking for COTS mobile solutions similar to those used by Jack Bauer on Fox TV’s “24” or the “ObamaBerry.”

The following sections explore using COTS hardware and software to create secure mobile communications devices. In particular, they focus on mobile security threats, lay out requirements for secure mobile devices, and present architectural options for integrating standard commercial smartphone device hardware with open source software and mobile virtualization. An Android proof-of-concept is also introduced.

Threats to mobile security

As mobile devices become increasingly similar to desktop and also data center computers, they suffer from the same maladies that plague corporate and government IT:

Vulnerabilities: Devices are subject to viruses, spyware, Trojan horses, and other malware. Application code bases and the middleware that supports them are large – tens of millions of line of code – and by definition untrustworthy and non-certifiable.

Buggy software: Mobile OSs and the software that runs on them are unavoidably buggy and thus are subject to a constant stream of zero-day attacks, which are exploits of application vulnerabilities unknown to the software developer. Open OSs supporting standard application development (such as RimOS and WindowsPhone) and open source OSs (such as Android or Linux) and applications that run on them emerge from commercial and community development of greatly varying quality and sensibilities – especially where security is concerned.

Brute force attacks: Mobile applications running on smartphones, including communications clients, are subject to Denial of Service (DoS) attacks on a per-process and system-wide level, for example. Of particular concern to government and other public entities are exploits that expose voice, text messaging, and other data streams to eavesdropping or other unauthorized third-party scrutiny. Another worry is the ability of unauthorized parties to interrupt mission-critical communications and spoof or forge participant identities and content.

A secure mobile communications device

Ideally, a secure mobile communications device such as a smartphone would be constructed from commercially available handsets and tablets that deploy off-the-shelf software platforms and applications running Android or Linux, for example. Devices need to support regular communications (voice, texting, social networking, and so on) and applications for “normal” conversations, but also enable secure exchanges (encrypted voice and secure texting/video transmission) among similarly equipped devices and/or infrastructure.

Ultimately, the scenarios of interest boil down to straightforward but difficult-to-meet requirements criteria. One requirement is that of secure communications, which refers to conventional applications including 3G voice, VoIP, Short and Multimedia Messaging Services (SMS/MMS), video, and so on, where clients run in secure contexts with options for encrypting data streams.

Open networks are also required. Using COTS hardware and software also predicates leveraging ubiquitous 3G, WiFi, and other public networks for private and secure communications. While GPRS, CDMA, 802.11, and so forth offer their own security regimes, those measures alone are insufficient to support secure and certified systems needs – they have been repeatedly demonstrated to be vulnerable to sniffing, spoofing, and other exploit techniques.

The requirement for off-the-shelf devices can also be difficult to meet. The vision for secure communications described herein builds on COTS hardware, but not necessarily mass-market handsets sourced through conventional operator and retail channels. Rather, secure communications devices represent collaboration of OEM handset manufacturers and third-party integrators and/or government contractors. Suppliers of aftermarket hardware encryption devices must also collaborate, such as vendors providing SD cards and/or encryption software running independently or leveraging existing capabilities in mobile chipsets. A concurrent requirement is that integration of these technologies still offers reasonable options for maintenance and upgrades, avoiding the expensive lock-in presented by legacy custom-built hardware and software.

Open software is also imperative. Smartphones and other devices suitable for the secure communications mission increasingly run open and/or open source OSs like Android, Linux, and so forth. As mentioned, a secure communications mobile device developer would be well advised to not replace those software platforms, but rather to augment or encapsulate them with secure and certifiable software.

Architectural options: Considering the candidates

Although this discussion emphasizes a COTS approach to secure mobile communications, it must also address the secure communications challenge holistically by considering several candidate architectures such as the following:

Point solutions

While requirements for mobile security are ubiquitous and system-wide, approaches to secure mobile communications tend toward single-function point solutions of limited scope in the software stack and communications stream. Such point solutions typically entail building and deploying secure and/or certified applications and middleware, encrypted data streams, and dedicated communications channels.

While constraining scope-of-effort is usually a good design and engineering practice, it ill serves modern mobile/wireless devices that are more like desktop computers than handheld radio sets. Unfortunately, these point solutions can still be compromised at an application level through cracking a saved or running application image, or by starving that application of compute resources (DoS).

Dedicated hardware

The most robust and secure method for isolating software is to run programs on separate pieces of hardware. Some mobile chipsets do feature unique companion processors to accelerate functions like graphics, video, audio, and in some cases, even encryption.

However, these dedicated coprocessors are configured as slave peripherals and do not provide sufficient context to run entire communications stacks and voice and messaging clients. In theory, more robust resources could be integrated on an SD card or other aftermarket interfaces, but then would lack means to transmit and receive secure data streams through an otherwise unsecured device without extensive modification to what should be a COTS OS and program stack.

Multicore architecture

High-end current-generation handsets and next-generation designs increasingly feature multicore application processors with two or more ARM processor cores. In theory, one or more cores could be dedicated to secure mobile communications, offering strong isolation from open OSs and open application environments.

In most cases, integrators or even OEMs would be hard-pressed to free up an entire CPU core for secure mobile communications processing, at least not without degrading overall device performance. (In a dual-core CPU, performance degradation could be up to 50 percent.) Moreover, even with a dedicated CPU core running secure communications loads, shared physical memory could still be subject to attack from the core(s) running an open application OS.

Virtualized architecture

A comprehensive and straightforward approach to architecting a secure mobile platform entails introducing mobile virtualization. This technology, like its data center cousin, runs over “bare metal” silicon to host an open application OS and software stack, or it could have one or more fully isolated secure cells (virtual machines) to host secure software in its own separate context. It could also have additional cells (as needed) to host selectively shared resources such as device drivers.

The benefits of a virtualized architecture are many:

• A very small trusted computing base (just the underlying microvisor) to ease certification and limit opportunities for exploits
• Deployment of existing off-the-shelf open OSs and software stacks in their own isolated cells
• Flexibility for integrators to instance new cells to meet the particular needs of a security regime and the technology to support it
• Capabilities-based security providing fully flexible dynamic protection management
• Defense against malware and security among cells through isolation and use of restricted intercell communications APIs – the visible open OS can become infected and fail without impacting secure communications software in other cells
• Fast Inter-Process Communication (IPC) mechanisms for high performance
• Resistance to DoS attacks through monitoring, prioritization, and load balancing among cells

Android proof-of-concept on the BeagleBoard

The mobile virtualization platform and security architecture depicted in Figure 1 builds upon a proof-of-concept announced by Open Kernel Labs last year. The OK:Android design for secure voice communications is based on Digi-Key’s community-driven BeagleBoard running a Texas Instruments OMAP3530 system-on-chip. It is detailed in a white paper available at www.ok-labs.com/products/whitepapers-abstract-page.

The secure voice design uses OKL4 to split up phone resources among multiple mobile hypervisor (or “microvisor”) partitions, including one for OK:Android, one for secure communications, and one for shared server compartments. The latter includes shared server cells for audio, various peripherals, system functions such as clocks and power management, and a console for debugging. OKL4 can also facilitate secure video and texting in other instances.

From proof-of-concept to deployment

Previously, securing government apps, voice, and SMS required costly custom hardware and proprietary software stacks. This resulted in one-off handsets and radios, saddling government workers with yet another device to carry, locking in their employers to a single vendor, and presenting IT support staff with platforms that are difficult to maintain and impossible to upgrade. Today, several major government integrators and their partners are developing and deploying real-world secure mobile devices by building on mobile virtualization and off-the-shelf mobile hardware and software.

Rob McCammon is Vice President of Product Management at Open Kernel Labs, where he is in charge of the OKL4 product line. During part of his 25 years of embedded industry experience, Rob was also Director of Advanced Technology Planning at Wind River Systems. He holds a Master of Management from Northwestern, and an MS in Computer Engineering from USC. He can be contacted at robm@ok-labs.com.

Open Kernel Labs 312-924-1445 www.ok-labs.com

CPU Tech, a manufacturer of secure processors, faced customer demand for different levels of functionality, operations, and security at different life cycle phases. As Vikram notes here, in the process of working with a third-party software licensing and entitlement management vendor, CPU Tech developed a list of criteria for meeting its customers’ licensing and pricing model needs.

With the recent surge in cyber security threats, it’s no surprise that military, government, and other sectors are eyeing secure processors and anti-tamper devices, which can help protect software and systems from reverse engineering.

In the military/defense technology community, projects and initiatives have extensive and complex government reviews and milestones. Therefore it is common for defense products and solutions to have a five-to-ten-year (sometimes longer) life cycle. Because of these long life spans, CPU Tech often found its customers did not need all product capabilities during all phases of a project. For example not everyone working in integration, test, or manufacturing needed to understand sensitive design details. Nor should everyone be allowed access to such details. To cite another example, there are times in a product life cycle when some security settings might be “locked down” for the remainder of the program. Also, some programs are “compartmentalized.” Compartmentalization gives engineers and users different access rights. This is all familiar territory for CPU Tech, which understood the various ways that its customers wanted to use and access CPU Tech processors. However, the company had no way to license, price, or allow access to support compartmentalization and other requirements.

Customers pressed CPU Tech for more flexible software licensing and pricing models. Needing an easy and efficient way to differentiate levels of functionality, operations, and security to customers at various stages of the product/project life cycle, CPU Tech began by identifying the following requirements for software licensing and entitlement management:

• Feature-based and role-based licensing and pricing models
• Provide embedded-node-locked and floating licensing capability
• Offer both offline (for machines operating in a classified area) and web-based activation
• Simplify complex product life cycle management
• Ability to automate the customer activation process

Ryan Kenny, responsible for product marketing at CPU Tech, explained, “With our Acalis Sentry Security Server solution we needed to offer our customers both feature-based and role-based functionality. Our solution can be operating in many different environments with different feature sets enabled or disabled, such as in development, manufacturing, security configuration, and security audit environments.” He added, “In each environment and use case, different people with different roles and security requirements operate the device. And in each use case, there are different feature sets enabled and disabled based on security and operations needs. For instance, customers may alternate between ‘call-back’ registration and standalone licensing and activation, depending on where they are in their development cycle.”

Early on CPU Tech determined it needed to engage with a third-party software licensing and entitlement management vendor. With its focus on its core competencies the company had no desire to develop a homegrown licensing solution, preferring to partner with an embedded software licensing and entitlement management provider. CPU Tech followed up on its initial list of criteria by describing key characteristics of the workable licensing approach:

• Appropriate and adequate cryptographic encryption for license key protection and storage
• Small memory footprint
• Support for processor architecture
• Support for embedded operating systems – needed to be OS independent, and easy to port
• Support for programming language
• Performance and reliability
• Easy to manage and track the license entitlement
• License activation automation
• Integration with other management systems, such as Salesforce
• Acceptable total cost of ownership

After researching its options, CPU Tech chose the FlexNet Producer Suite for High-Tech Manufacturers from Flexera Software. Ryan noted, “Flexera Software’s embedded software licensing and entitlement management solution enables CPU Tech to protect our intellectual property and allow the customer to operate in a classified/secure area without Internet access. We needed a solution that enables a solid revenue model without expanding our manufacturing costs.”

“With Flexera Software [see Figure 1] we can now easily upgrade and downgrade our customers without deploying additional hardware as well as offer them licensing and pricing models based on roles and features,” he added.

Embedded licensing technology and entitlement management will enable CPU Tech to realize a reduction in manufacturing costs and to better manage its Acalis product life cycle. Now CPU Tech can offer a single version of its hardware and simply control the hardware capabilities and features based on roles and features. In addition, the licensing adds a valuable security layer in user activation, making sure only those entitled are able to activate the product.

“Flexera Software’s embedded licensing technology enable us to protect CPU Tech and customer intellectual property,” Ryan commented, adding that “with its entitlement management solution we expect to see clear savings in the operations area by being able to easily upgrade/downgrade capabilities on the hardware. We will be able to bring products to market faster and tailor subscription licenses to meet customers’ unique requirements, helping us to better serve our customers.”

For CPU Tech, flexible software licensing and entitlement management allows for cost reduction and revenue models that matches customer needs and processes. In the past, much of what were security “rules” to be enforced through audit are now enforced by embedded licensing and entitlement management.

Vikram Koka is the vice president of research and development at Flexera Software, where he is responsible for engineering for the FlexNet Producer Suite for High-Tech Manufacturers. Prior to joining Flexera Software, Vikram was the chief architect for Macrovision across all its product lines, including the DRM, video, and game distribution offerings, in addition to software licensing. Vikram has held senior engineering or technical management positions at large companies such as Silicon Graphics (SGI), and LSI Logic as well as startups such as CellMania and SmartDB. Prior to Flexera Software, Vikram’s background included mobile, ecommerce, enterprise applications (both hosted and on-premise), databases, and distributed applications. Vikram has an MS in electrical and computer engineering from the University of California at Santa Barbara, and a BS in electronics and communications engineering from Osmania University, Hyderabad. Vikram currently has four patents pending and was one of the early contributors to the GNU Debugger (GDB), a popular open source debugger.

Flexera Software flexerasoftware@eastwick.com www.flexerasoftware.com

Stephen covers key system software issues that embedded systems developers must address, including next-generation SoCs that contain multiple cores and methodologies to properly allocate the applications between several types of operating systems.

Embedded developers face several decisions when developing medical embedded devices, from selecting the best system software for optimal application performance, to understanding the interactions and limitations between the software operating system and target hardware. Should the software engineer use a small micro-kernel, Real-Time Operating System (RTOS), or a General Purpose OS (GPOS) such as Android or Linux? Other considerations include the physical size of the system for portability and functionality requirements, including faster performance, power consumption, data protection, and display (user interface) technology. And FDA certification and industry standards that affect embedded software selection come into the mix as well.

Modern medical devices are evolving at a record clip. From portable wireless units for patients to use at home to larger more complex devices used by healthcare professionals at a facility, there’s no question we are at the forefront of developing new ways to empower patients and medical professionals alike. How do we make sure the system software that controls these devices does exactly as planned with little to no risk of harming the patient?

At the heart of the matter: operating systems

Typically an Operating System (OS) manages a medical embedded device. An OS can vary from a simple “roll your own” built in-house by a few enterprising software coders to a more complex OS from an established vendor. A GPOS such as Linux or Android establishes a feature-rich platform for application development, but sometimes consumes more memory than necessary. An RTOS is also a good choice for modern medical devices, particularly when specific system requirements require a deterministic preemptive kernel and a small memory footprint. Somewhere in the mix there is an ideal candidate for your application and hardware. One thing is certain: Before selecting your OS, know exactly the intentions of your application and the hardware you plan to use.

How will the device be used?

One way to minimize risk when developing your embedded system is to first consider its use cases – not just how the end-user will interact with it, but also how it will be designed, developed, and tested. Will the device be used primarily by a healthcare provider, by the patient at home, or both?

Does the device have communication modes or is it purely a stand-alone? Depending on its communication needs, you may find that your preferred OS includes many of the modes you need, or you might prefer another OS, in which case you’ll have to port over the communications stack and/or the driver to attain the right mix of communications software.

Are there any real-time needs identified? For some devices, there is no requirement for real-time behavior. If an interrupt is serviced 100 milliseconds late, the results may be delayed by 100 milliseconds, but that’s not going to cause a failure. However, if it’s a laser involved in eye surgery, this can have catastrophic effects if the laser does not turn on and off at the precise time. If the laser has eye tracking guidance, the laser must move in lockstep with a predefined pattern even in the presence of eye movement.

Perhaps the device is a critical piece of equipment, so there is minimal sensitivity to cost. On the contrary, a device that is handheld and sold in the millions has a high sensitivity to cost. These types of considerations will directly affect the need to minimize BOM, which in turn results in possibly minimizing the memory you’ll need to effectively build the complete application with some margin.

It’s all about the hardware

Once the use cases have been defined, it’s time to find the appropriate hardware. Medical systems can be extremely small, with an 8-bit microcontroller clocked at less than 25 MHz, and use only 8K of memory. More complex designs can include feature-rich SoCs clocked in the hundreds of MHz and megabytes of memory. The range of systems encompasses hybrid systems that have special purpose processors or DSPs to systems that include numerous multicore chips.

What’s best for your design comes out of the use cases and expectations on how you want the system to behave.

Is multicore necessary?

To the two main reasons that come to mind for selecting multicore – pure processing performance and low power management – a third could well be added, the combination of the two.

If you’re concerned with low power you may want to use a multicore SoC simply because it can utilize all the available cores at a lower clock frequency rather than clocking a main processor at a much higher frequency. When not needed, it can power down the extra cores to save power.

While both power and performance are good reasons to use multicore, the question is more about finding the best way to allocate the CPUs. With symmetric hardware you can use a single operating system across all the available cores as a type of Symmetric Multicore Processing (SMP). Most GPOSes and some RTOSes have this capability. Using SMP could complicate the scheduling across the cores, however, as real-time hits due to cache misses in one core could cause a cache flush in the other core, which invariably leads to delays in the system. Features such as spinlocks are common to all SMP-capable operating systems. If not employed correctly, a spinlock can hurt system performance, as one core stalls for an indeterminate time waiting for the resource on another core to be freed.

The other way to build the system (even with symmetric hardware) is to apply Asymmetric Multicore Processing (AMP) techniques. This approach involves two or more separate operating systems (Figure 1) interacting through some type of communication channel using hardware like a series of FIFOs or through shared memory. There is a standard that makes the application development portable by the Multicore Association called the Multicore Communications API (MCAPI).

When hardware and software worlds collide

Consider the case where a medical device that has USB connectivity to a Windows host computer usually follows the USB specification, but when all the parts of the SoC are activated, the hardware intermittently starts signaling outside the specification in such a way that the host computer shuts down the port in the middle of a session causing failure at the most inopportune time – during patient data collection. With results lost, the patient, who made special preparations for the original procedure 24 hours in advance, must prepare for retesting.

Two fundamental reasons caused the port shutdown. First, the software assumed that the USB controller would not fail. And second, the system architecture had not planned for the case where the unit is unplugged in the middle of a session. Had the system taken either one of these use cases into account, the system would have stored the data locally, allowing for a transfer after the session was re-established, thus minimizing the risk to the patient of a possible retest.

The application relied on the USB controller’s flawless operation to prevent data loss. If the application had been broken down into several sections, data loss might have been avoided. With an architecture where data collection and data transmission are not interrelated, even if the link goes down, the data is still stored in the device, so when it recovers it can pick up where it left off and not lose any data. A write to a buffer before the transmission to the host, which would occur in the background, is one way to avoid lost data in this type of scenario.

If a software workaround to detect the USB bus suspension is used, the workaround can take the SoC pins out of USB mode and make them GPIO pins so that the host can detect a reset condition and force a re-enumeration of the device. The USB software would then resubmit the buffers, and the transmission would resume. The end result is that the data would not be lost, just delayed while the workaround took place.

Portability considerations

An OS manages the system’s resources both in hardware and software. The most basic management is that of memory and time. But where does the responsibility of the OS stop and that of the application start? While an application can have a device driver built into it and talk directly to the hardware, porting to new hardware becomes a challenge as the device evolves and newer hardware is employed. Therefore, it is recommended that most, if not all, devices in the system be managed by the OS in order to ensure future portability.

Regulations and patient privacy

The need for portability includes wireless devices like a GSM radio or an 802.11 wireless interface for connectivity. Others include Bluetooth and ZigBee, where these links must also be secure and provide patient privacy. Even in the device itself, it’s imperative that only doctors who are authorized to see a patient actually see the data of that patient. Disallowing unauthorized access is also a critical requirement of any device. Are the records secure, even for the technician who works on the equipment? Are there any modes where this data is not secure? True Health Insurance Portability and Accountability Act (HIPPA) compliance is making sure that the security of information is paramount. Security inside the database of patient records is as well.

Conclusion

Medical devices are a special breed that will touch all of us in some way. We need to take extra care when designing these systems to ensure that the device does what it is intended to do. Does it make sense to use an RTOS or a GPOS to meet the requirements of determinism, size, boot time, power optimization, and the breadth of middleware available? Finally, to minimize risk we need to make sure that all regulations with both HIPPA and the FDA are followed.

Stephen Olsen has over 20 years of embedded software experience with his past 15 years spent at Mentor Graphics. Stephen is currently technical marketing engineer for Mentor Embedded, Mentor’s embedded software division. During his tenure at Mentor, he has co-chaired VSIA’s Hardware dependent Software (HdS) design working group, worked on the MRAPI specification for the Multicore Association, and authored several papers on system design and power management. Stephen has worked in consulting, system architecture, embedded software, and IP. He holds a BS in Physics from Humboldt State University in California.

Mentor Embedded Stephen_Olsen@mentor.com www.mentor.com

In a simple view of the world, an application running on a multicore system should run at least as fast as the same application runs on a single-core system with the same CPU power. Unfortunately, in practice, that is not the case. However, by implementing application parallelism and using an SMP OS and an embedded hypervisor, one can solve the challenge and realize dramatic performance improvements.

Let’s face it. The Multicore Era is upon us. How did we get here? For years, processor manufacturers delivered on the Moore’s Law promise of doubling CPU performance every couple years by increasing the number of transistors, increasing clock rates, and increasing instruction-level parallelism. We saw clock rates go from 1 GHz in the year 2000 to 2 GHz in 2001 and finally to 3 GHz in 2002, where we seem to have hit a clock-speed brick wall.

The combination of increasing power demands and rising chip temperatures seems to have put the brakes on the clock speed race. One of the surest signs was Apple’s switch from the PowerPC to the Intel Architecture. Promises of 3 GHz PowerPC G5 processors in Apple laptops never materialized due to power and heat problems. Processor manufacturers quickly realized that to keep doubling performance, they needed a new trick. That new trick was to add multiple cores to a chip. Now Mac computers with 12 cores at 2.93 GHz exist, and coincidentally, dual- and quad-core single board computers and systems are very prevalent in the military embedded realm today.

However, if the software running in the system is not optimized for multicore, there can be degradation in performance when migrating from a single-core based system. The higher transistor density in a multicore CPU does not generally translate to an increase in speed on applications that are not parallelized. Additionally, SMP OS and embedded hypervisor RTOS technologies can aid optimum performance in single-core to multicore migration. But first we will examine the issues of synchronization, concurrency, and scheduling.

Synchronization and concurrency overhead kill performance

A real-time embedded SMP operating system must maintain deterministic scheduling and interrupt response as well as respond rapidly to interrupts and high-priority tasks. This job becomes substantially more difficult and time consuming in a multicore CPU. For example, when multiple CPUs are active, data may be accessed by more than one of them simultaneously, adding a new level of concurrency issues for the OS to deal with. This requires additional mechanisms for concurrency control. Because of the increased concurrency in multicore systems, locking and synchronization mechanisms are more complex and take more CPU time than in single-core systems.

Generally, on a single-core system, a critical section of code can be protected from interrupts by simply disabling preemption, doing the necessary work, and re-enabling preemption. Enabling and disabling preemption can be as simple as storing a value in a variable. In a multicore system, each core has its own unique set of interrupts, so disabling preemption does not make a lot of sense, since code on the other cores could still execute the critical section. Instead some form of locking needs to be introduced. In LynxOS, a deterministic hard real-time OS from LynuxWorks, this is done with Kernel Spinlocks and requires the use of special hardware mechanisms such as locked bus cycles. These mechanisms are considerably more complex than the simple memory accesses a single-core system uses, and this complexity adds to overall performance degradation.

In addition, when code on one core is in a critical section, code on other cores is blocked waiting for the code to finish the critical section. If the locks are coarse-grained, it is possible that several cores could be idle because they are unable to schedule any useful work.

The synchronization and concurrency overhead incurred on multicore systems is most visible at the operating system software level, but is also apparent in multi-threaded applications that rely heavily on constructs like condition variables, semaphores, and message queues. Operating systems compiled to support multiple cores are typically about 10 percent slower on a single-core system than the same operating system compiled to support a single core.

Scheduling threads across multiple cores

The scheduling algorithms play a key part in harnessing the power of multiple cores and can cause performance issues if not implemented carefully. Typical scheduling algorithms maintain a per-CPU queue of threads that are ready to run and allocate CPU time based on this queue. However, in a real-time system, it is critical to preserve real-time determinism, so the scheduling approach is different. The scheduling happens on a global basis where the highest-priority thread runs on the first available CPU. However, this may lead to higher levels of cache misses. This can be addressed by using design optimizations in real-time thread scheduling.

One such design optimization, known as processor affinity, allows applications to request an “affinity” to a processor core. In this case, the operating system schedules the applications on the preferred processor core, as long as it does not affect overall system scheduling. A more rigid form of processor affinity is processor binding, where the task is always scheduled on the same processor core. However, this approach in RTOSs may lead to priority inversions. Operating system design should accommodate considerations such as processor affinity without degrading real-time determinism and responsiveness. In the context of a real-time operating system, other key factors such as priority scheduling and interrupt latency should be preserved in multicore architectures.

An SMP-enabled real-time operating system must schedule tasks dynamically and transparently between processors to efficiently balance workloads using available processors. It optimizes the support of load balancing on multiple cores along with preserving the key elements of real-time latency and determinism. If the operating system “bounces” the application from core to core, the application will take additional Translation Lookaside Buffer (TLB) and cache misses, reducing performance. On the other hand, if the application is “pinned” to a core, there may be enough additional demand placed on that core to slow down the application, compared to running it on a single core.

Taking full advantage of multicore processors

To maximize multicore performance, application parallelism, SMP-enabled OSs, and embedded hypervisor technologies should be explored.

Application parallelism maximizes CPU utilization

All applications should be carefully examined for opportunities to parallelize the tasks. In parallel computing, an application is broken down into threads that execute independently on separate cores (Figure 1). Application parallelism is dependent on the ratio of computation to communication overhead. The computation is the amount of time the CPU spends executing application code. The communication overhead is the amount of time that the OS spends in communicating between cores. In a typical multicore architecture, the communication overhead indicates how often messages are sent between different cores. The more threads an application has, the higher the chances that they are scheduled on different cores, which in turn increases the communication overhead.

Each type of system has different characteristics, but when optimizing application parallelism to maximize performance, there are broadly two types of application parallelism that can be used:

1. Coarse-grained parallelism is characterized by large tasks, single threaded and low communication overhead. In this case, the ratio of computation to communication overhead is high. This indicates that the communication overhead is lower than computation time, thereby yielding better multicore performance.

2. Fine-grained parallelism is characterized by small tasks, multithreaded and high communication overhead. In this case, the ratio of computation to communication overhead is low. This indicates that the communication overhead is higher than computation time, thereby yielding lower multicore performance.

Applications that are CPU-bound can exploit the full power of multicore architectures since they are coarse-grained, while memory-bound or I/O-bound applications (fine-grained) may need to be optimized to avoid the bottlenecks that arise due to the communication overhead in symmetric multiprocessing architectures.

POSIX-based OSs provide a rich environment of threading functionality to make it easy for developers to implement parallelism in their applications. Developers must consider the design trade-offs of using multithreading versus non-multithreading to harness the power of multiple processor cores. In some instances, applications may perform better on a single-core system.

Multicore optimization with SMP OS and hypervisor technology

Another approach to multicore optimization centers around choosing an appropriate OS. An SMP-enabled OS can help add concurrency to an application by balancing the threads running on multiple CPUs and maintaining a deterministic hard real-time performance level.

But what if you could get even more control over how the OS runs on the multicore CPU? A new trend emerging in multicore environments is the use of a small hypervisor operating system, which abstracts the capabilities of hardware and allows multiple heterogeneous operating system instances to run on a single hardware platform. A Type 1 hypervisor, such as LynxSecure from LynuxWorks (Figure 2), runs directly on the hardware and has complete control of the platform, providing superior utilization of processor resources. In the SMP-enabled hypervisor, a single copy of the hypervisor can allow a single guest operating system to utilize multiple cores. The same hypervisor can enable AMP by allocating a single guest operating system to a unique core. This can be extended to allow AMP and SMP on the same platform through judicious allocation of guest operating systems on single or multiple cores, thereby increasing processor utilization significantly.

John Blevins is the Director of Product Marketing and Tools Development at LynuxWorks, with more than 25 years of software experience in the embedded industry. Contact him at jb@lnxw.com.

LynuxWorks 408-979-3900 www.lynuxworks.com

While the Android SDK provides a great starting point for an individual developer of Android code, it is missing features that facilitate the collaboration and coordination needed when a team is developing an Android application. By integrating the device-specific, native platform SDK with a compatible commercial development solution, agile teams can achieve tremendous efficiencies and higher-quality results.

The Android Software Development Kit (SDK), which Google provides for free, is a great starting point for developing an Android-based smart device application. The SDK contains a variety of useful materials for developers, including extensive documentation, tutorials, samples, best practice guidance, and an array of tools for numerous development purposes.

The SDK’s set of Java APIs gives application developers access to native functions that Android-based devices support, such as 2D and 3D graphics, multimedia codecs, telephony features, and location services. A device emulator in the SDK allows developers to try out their code directly from the development environment without requiring a physical device. And the SDK has an Eclipse plug-in that exposes the Android APIs and SDK tools in a rich Integrated Development Environment (IDE).

Opening the door to collaboration

For an individual developer of Android code, the SDK is valuable and is becoming more so as it is being extended with new features all the time. However, it is missing features that facilitate the collaboration and coordination needed when a development team is creating the application.

By integrating the device-specific, native platform SDK from Google with a compatible commercial development solution, agile teams can achieve tremendous efficiencies and better results. Integrating the native Android SDK with a commercial development environment opens the door to seamless source control, iterative application planning, effortless work item management, and a host of enterprise-quality development capabilities for an Android application.

For instance, many Android applications are structured as hybrid Web applications, where part of the application runs on an application server on the network delivering data to the device from an enterprise storage system, perhaps a mainframe computer. Another part of the hybrid application runs on the device itself, displaying the data it receives across the network and formatting it for the device form factor while accessing the device’s services such as GPS, camera, and accelerometer to deliver a rich and well-performing user experience.

Such a hybrid application is typically created by a small team comprising a few developers of the fundamental business logic and Web application components, a few User Interface (UI) developers, a user experience designer, a couple of testers, and a team leader or manager. Let’s consider how this team can leverage the Android SDK in an environment that allows each member to efficiently communicate and collaborate.

The integrated Eclipse environment

The Android SDK, or more precisely, the Android Development Tools Eclipse plug-in that is part of the SDK, can be combined with an Eclipse-based commercial collaborative development product such as IBM Rational Team Concert (RTC). There is much information available that documents how to get the Android SDK and RTC working within the same Eclipse “shell.” See Figure 1 for an illustration of what the Android SDK looks like when integrated with RTC.

The commercial IDE (RTC) offers integrated work item management, which allows the team leader to define work for the Android UI developers and assign those tasks to them, separately from the work assigned to the application logic developers and the other team members, including work assigned to the testers. The code changes associated with a particular work item are tied together into a specific change set that is delivered in one shot, so the full code change can be tracked as a unit. As the developers edit files inside their IDE, the change set is automatically maintained. The developers don’t have to do anything special to produce the change set other than edit the files they need to work on.

The change sets can be shared amongst members of the team before being fully integrated with the main code stream. So a change set altering the format of data supplied by the Web application can be shared with the UI developer working on the logic that displays the new data without affecting the rest of the team. Once both UI code changes and Web app code changes are deemed ready, they can be integrated in one synchronized task into the mainline code stream for the rest of the team to use.

Developers working on the Web application can execute the part of the application that runs on the device from their own IDE using the Android device emulator that is part of the SDK. Using the combination of shared change sets and an integrated device emulator, the pairs of developers working on the same feature (UI and Web app logic) can collaborate to work out initial problems that may arise as the result of different understanding about the application details. One of the developers can capture a screenshot of the device emulator using the screen capture tool built into the commercial IDE and share that screen capture with the other developer to show an exact behavioral issue or defect in the code.

Agile team collaborative development tools such as RTC allow the definition of multiple short iterations where a small set of application enhancements are to be implemented and validated. A typical agile iteration is two to four weeks long. The team leader can work with the team to map work items from a backlog list into the specific iterations and assign the work items to individual developers. As the developers pick up the work items and begin to make progress on them, their effort is automatically recorded and available for the team leader to track and view. This makes the information about what has been completed, what is being worked on now, and what is still to be done easy to track and view in a dashboard presentation. Everyone on the team can see how the iteration is progressing and the status of the work items planned for that iteration.

When the testers on the team start the functional testing of the application, they can open defects as work items in the shared development project. They can easily grab screen captures of the failed tests and include them in the defect records. The team leader can track these incoming test defects and work with the team to distribute them for resolution.

Products add value to basic SDK

Several Eclipse-based commercial products can be integrated at the same time with the Android SDK to provide the team with even greater capabilities. For example, the ability to model the device code structure and keep this model in sync with the real source code can be added to the collaborative agile team environment. By integrating a commercial product such as IBM Rational Rhapsody with the combined Android SDK and RTC environment, the team can gain the ability to keep a high-level model of the application in sync with the actual application code. Because the real application structure can be difficult to understand for moderate-sized projects, the ability to generate a model from the source code can prove very valuable for the team.

Commercial static analysis products can be integrated with the Eclipse-based development environment and deliver the ability to analyze the code for quality and security issues. Some of these products can be integrated with the actual change set deliver process so that no code is integrated into the main line code stream unless it has been analyzed for fundamental quality and security issues.

All of the capabilities delivered by the commercial development products extend and enhance the basic SDK supplied by Google. While the Google Android SDK is the fundamental starting point for any project delivering code to be executed on the Android platform, the SDK can be dramatically more effective when integrated with the traditional agile team development features available in other Eclipse-based commercial products becoming more widely available today.

Leigh Williamson is an IBM Distinguished Engineer who has worked in the Austin, Texas lab since 1988, contributing to IBM’s major software projects including OS/2, DB2, AIX, OpenDoc, Java, Component Broker, and WebSphere Application Server. He is currently a member of the IBM Rational Software Chief Technology Officer team, influencing the strategic direction for products in the Rational brand and leading the projects for software development automation and mobile device application development. Leigh holds a BS in Computer Science from Nova University and an MS in Computer Engineering from the University of Texas at Austin.

IBM Rational leighw@us.ibm.com www.ibm.com

As discussed in our recent E-cast on “Enabling Telehealth Devices” (archived at ecast.opensystemsmedia.com), the FDA is recommending static code analysis for the medical device software development process to detect problems before release. Here’s a look at best practices for utilizing static code analysis tools.

Medical devices are using more software code than ever before. However, while software provides medical devices with significantly more capability and flexibility, it also brings additional complexity, which translates into increased risk of failure. About 20 percent of medical device recalls today are caused by software defects, and the number is rising.

The Federal Drug Administration (FDA) oversees the quality of medical devices sold in the United States, and companies wishing to release a medical device must receive FDA 510(k) clearance. While post-market failures are investigated, the FDA is putting stronger focus on prevention and recommending that static code analysis be used as part of the approach.

The value of sophisticated defect detection

Modern static code analysis tools use sophisticated techniques to analyze source code to detect potential software defects. Tools try to analyze all logical paths in the code, providing significantly more path and code coverage than traditional forms of testing. Static analysis tools do not require any test cases and can operate even on fragments of code, finding potential program crashes, buffer overruns, memory leaks, data corruption, and more. Static analysis usually operates quickly and can report a range of potential bugs in a relatively short amount of time (see Figure 1).

For a variety of reasons, static analysis tools do produce some erroneous results, generally called false positives and false negatives. A false positive occurs when the static analysis tool believes there is an error when there isn’t. A false negative is when an error should have been reported but isn’t.

Most modern static analysis tools must perform a delicate trade-off between finding as many good results as possible within an acceptable accuracy level and an acceptable running time. Stated another way, noisy tools that find every problem in a sea of false positives may be of limited value, as are highly accurate tools that find only a small class of issues (see Figure 2).

Modern static analysis tools have improved analysis techniques to generate useful results at an adequate level of accuracy. Most organizations recognize that static analysis tools, while imperfect, provide significant value in most any software development process.

Making the most of static analysis tools

Modern static analysis tools are relatively new to most medical device manufacturers. For many organizations that are instituting static analysis in their process for the first time, knowledge of best practices can help get the most out of tools in the shortest amount of time and with the least amount of rework.

Tuning

Static analysis tools are delivered with generic settings applying to all types of code bases, and while they find good bugs right out of the box, results can be improved greatly just by tuning the tools for the code (see Figure 3). This helps find more relevant bugs and reduces the search through false positives, which wastes time and causes developer fatigue.

Many static analysis tools have their own source code parsers that might not understand or have access to all of the code. Configuring the system to analyze all of the code or tuning the system to recognize interfaces inaccessible at analysis time – such as separately verified third-party libraries – ensures the results are optimum and repeatable. Achieving 100 percent code coverage is important to close holes that can increase risk.

Tuning helps uncover real problems. For instance, telling the static analysis tool how a memory allocation mechanism works or when a program exits so that the tool isn’t continuing to track issues along a certain path can help uncover new problems and prune away false ones. This can be a tedious process requiring specific expertise, but it pays off in the long run.

Tuning is typically an ongoing process and should be reviewed periodically to ensure that configurations are used consistently and keep pace with the changes to the code and the environment. Without ongoing tuning, developers will likely miss some important bugs and the team will waste time scrutinizing false positives.

Configuring checkers

Many static analysis tools come with hundreds of checks that cover a range of issues, from concurrency to security to C and C++ pitfalls. Many do not necessarily apply for a given application. For instance, why turn on a C++ specific check when analyzing C code? Determining the right set of checkers requires some trial and error as well as expertise to understand what is going to give the best bang for the buck. Some areas for consideration are: what categories of checkers can result in real problems, which checkers tend to be noisy, and which checkers can be configured to be useful. Once a good set is finalized, lock it down so it is documented and runs consistently.

In a real-world example illustrating the value of checkers, a customer wanted to ensure that their static analysis system was running consistently and requested to be alerted immediately if there was a discrepancy in the system. Developers created a test suite that included test cases for each checker. Whenever they changed the system, they ran the test suite to ensure that every checker was indeed operating as expected. If the results failed, they knew they had a configuration problem that needed to be addressed. If the tests passed, developers dropped the results into their design history file as proof that the system was working as they had documented. This test suite not only gave the customer accountability and assurance, but also decreased their maintenance and administration costs.

Process

Once full coverage is achieved, the system is tuned, and the breadth of analysis is defined, developers can begin using static analysis much more effectively. For medical devices, a typical goal is to examine every single issue reported. Each issue can be categorized in a number of different ways:

• An issue that must be fixed. It will have an appropriate priority assigned that describes its importance and how it must be addressed during the software development process.
• An issue that is correctly flagged, but not likely to manifest as a real-world bug, usually because there is an incorrect environment assumption made by the tool. These types of categorizations signal a potential tuning opportunity.
• An issue that is incorrectly flagged as an error, either a false positive or an outright bug in the analysis tool. These issues also signal a tuning opportunity.

Each of these cases must be carefully reviewed. False positives in particular should be examined for correctness. Liberal documentation is required for each issue, and a robust data retention policy is necessary for full accountability. These triaged defect reports will likely be revisited either in an audit process or in a retrospective if a major bug is found later in the process. It’s common for organizations to go back to the static analysis defect to see how a major bug got through the process. It could signal a broken process or an opportunity to tune the analysis to find better bugs.

Usage model

Static analysis is typically run either in a developer sandbox build and/or through a central build (see Figure 4). At a minimum, analyzing and evaluating the results makes sense to do just before release. However, software development organizations shouldn’t wait to the last minute to address a potentially large pile of bugs, particularly when those bugs could have been addressed earlier as part of a disciplined process. Otherwise, teams risk missing a deadline and changing the code at the worst possible time.

Organizations often automate static analysis as part of a nightly build or a continuous integration build. In this way, results can be reviewed frequently and addressed as they arise. Others perform the bug-finding process earlier by enabling developers to analyze the code they are working on right in their sandbox environment. Developers can get immediate feedback on the quality of their code changes and then fix and verify defects before check-in. The quicker the cycle time, the cleaner the code will be in the repository.

Regardless of where it runs, the technical environment needs to be consistent to ensure that the results are the same. Central and developer builds need to be consistent. A slight change to the analysis settings can result in many more results being reported, and organizations don’t need the added burden of having to review more problems that might be mostly false positives. Creating a highly automated system for developers to use will help ensure consistency.

Many medical device companies check not only the source code into their repositories, but also their actual environment. In this way, traceability is available. Static analysis executables and all the associated configurations, states, and other relevant items should also be checked in regularly to ensure consistency and accountability.

Dealing with backlog

Most organizations begin using static analysis after a significant amount of code has already been developed. Generally, the more code there is, the more bugs will get reported. Thus in rolling out static analysis, management must allocate upfront time to deal with the initial backlog of bugs.

It’s better to try to institute static analysis as early as possible in the development cycle to minimize the backlog, then create a process to deal with the backlog separately from the daily flow of incoming bugs due to day-to-day code changes. Reviewing defects takes time and should be allocated properly among the developers or farmed out to a separate team to pick the defects needing work.

Culture

All development teams are heterogeneous in technical skill levels and in how each individual within the team defines quality. During training and mentoring sessions, the most common arguments are:

• “Yes, this is definitely a bug, but the code has been working so we don’t want to change it.”
• “We should not allow code like this to be in our product.”
• “This scenario would never happen in real life.”
• “This will become a bug if we port the product to another platform in the future.”
• “If you spent just a few more minutes on this, you’ll see it’s clearly a bug.”

Static analysis will deliver bugs of all stripes, from critical must-fix problems to warnings. Some organizations want to be opportunistic and only change the code for provable bugs. Others proactively clean up code and improve quality, even going so far as to “fix” warnings. Teams should have consistency in how they address static analysis results. Review of the results, training/mentoring, and frequent communication are keys to success.

When used correctly, static analysis has proven itself to be highly effective in improving software quality for safety-critical code. Although not strictly required for approval, the FDA recognizes its efficacy. With the proper planning, expertise, and realistic investment, static analysis should yield a significant return on investment and help deliver safe code to the marketplace.

Andrew Yang is managing director and cofounder of Code Integrity Solutions. He is former VP of products and services at a leading static analysis provider. Andrew has a Computer Engineering degree from Brown University and an MBA from the MIT Sloan School of Management.

Code Integrity Solutions

andy@codeintegritysolutions.com

www.codeintegritysolutions.com

Managing the media in a device calls for a database, but several design issues need to be addressed to make both the consumer experience and the developer experience as good as possible.

Portable media players offer new ways to deliver and access content. Limitations that device manufacturers have been able to take for granted, such as amount of content and delivery method, are gradually eroding. In addition to metadata about media stored on the device, such as artist, album, and song names, players with network connectivity can map recommendations from friends. With increasing storage capacity and the new possibilities that come with ubiquitous connectivity, portable media players are set to handle more data than ever before.

As with any consumer device, cost remains a critical factor in portable media player design. Software developers must create new techniques to manage music, video, and image files that leverage the limited hardware available for these products. The amount of data that must be managed can easily exceed a device’s working memory, posing unique challenges to developers who want to access that information for a variety of purposes.

Software developers can turn to relational embedded database management technologies for the scalability required by modern portable media players. A database file stored on inexpensive NAND flash memory is a simple solution because the details of storage and retrieval are handled by algorithms with proven scalability and reliability, adapted to the unique requirements of portable devices. Design engineers can capitalize on this advantage to reduce cost and time to market.

Organizing media files

There are many ways to find a song on a portable media player. A user might look up the artist and album, then select the song from the album’s track list. Or the user might search for the song by title after browsing through an alphabetical list. Sometimes the song is chosen randomly.

To do this, the media player must have a list of all songs stored on the device. This list can be generated when the player is started or when the song is transferred to the device if a media-aware protocol such as Media Transfer Protocol (MTP) is used. This list should be stored persistently so that it is not recreated each time the device is started, and should allow the data to be browsed in several different ways. If songs are added or removed, it should not be necessary to rebuild the entire list.

Hardware limitations

Portable media players are designed for a single task: playing music and video. These devices typically use dedicated DSP hardware to decode media, leaving the general-purpose CPU free to operate the user interface and send media to the DSP. Unlike a desktop application, where a rogue task can sometimes consume all CPU resources, media player software is carefully designed to limit the amount of time spent on each task so that media playback is never interrupted.

Media files and their metadata are stored on NAND flash memory or miniature hard drives. While these are inexpensive and persistent, both types of storage are divided into blocks that affect performance. Metadata is accessed randomly, so data is often read into RAM far in advance of when it is needed. For this reason, it is necessary to cache frequently accessed blocks so that these extra reads are not wasted.

But because available memory is limited, media player software must use a predetermined amount of RAM for general processing. The total RAM required for a software component, such as a lightweight relational database, is based on four measurements: static code, static data, stack, and heap. The exact amount of RAM required, or footprint, depends on which parts of the software are actively running at any given time.

Embedded relational database

When software developers implement a custom data management framework for a portable media player, the data model is usually hard-coded in the framework itself. Changes to the data model must be made carefully so that worst-case guarantees are not violated. The alternative, developing a generalized data management framework, is difficult to justify for a single product.

While a traditional database management system requires too much overhead to be used on a portable media player, the core database technology and algorithms are still valuable for portable media players. An embedded relational database provides a standard interface for reliable data management that is suitable for media players and other portable devices. The database is invisible to the consumer and requires no administration because it is linked directly into the application as an in-process software library.

Database technology provides applications with a practical model for data storage that is powerful yet easy to use. Information is organized into tables with a predefined structure. By hiding the exact format used to store information, the database can cache information intelligently to optimize performance while providing a wide range of features – transactions, recovery, search, and shared access – that would otherwise be difficult to implement and maintain in a single application.

Application code is frequently reused to expand an existing product line or to jump-start development of a new product. Using a database gives an application a consistent architecture for persistent data storage, making it easy to add new features and migrate the application code to a new environment. Using a database is a good long-term investment because it lets applications scale through the entire life cycle of the application.

Portable media players need consistent, scalable performance across all operations, whether reading or writing to the database. Indexes are used to efficiently search the database and traverse the relationships between tables. B+ tree indexes are optimized to minimize disk I/O and offer consistent performance regardless of table size, even with limited RAM.

When a sudden power failure or crash occurs while writing to a file, data corruption and inconsistency can result. To prevent corruption, embedded databases first write each change to a separate log file before modifying the database file. Using the log, incomplete changes can be rolled back to restore the database to a known good state.

Rollback helps application developers cope with errors. If the software on a media player were to encounter a media file that is partially corrupt, it might have already added data about the file to several tables in the database. Rollback lets the application group changes together into an atomic transaction that either finishes completely or not at all. When an error occurs anywhere in the transaction, the application can roll back all changes with a single API call, regardless of where in the process the error occurred.

Solving media player storage problems

Nearly every consumer now owns a portable media player in some form, and the demand for more storage capacity in these and other embedded devices continues to drive down the cost of storage hardware. But new hardware capabilities present new problems to media player software.

ITTIA DB SQL is a pure relational database library suitable for portable media players where performance matters (see Figure 1). No matter how large the collection of media files grows, media will be sorted, searched, and navigated rapidly because the database library maintains indexes on key fields in the metadata. Interoperability, communication, and synchronization with other consumer electronics are areas of expansion for application developers. Device applications embedded with ITTIA DB SQL can leverage row-level locking to add concurrent access to the metadata for other devices. The database library exercises tight control over memory and code footprint and stores data in a portable format so that media player developers can face current hardware restrictions while providing for the needs of the future.

Sasan Montaseri is the founder and president of ITTIA. He has more than 20 years of experience in the embedded database business arena. He holds a BS in Electrical Engineering from the University of Kansas with a minor in Mathematics. He was a member of PI MU Epsilon National Honor Society of Mathematics.

Ryan Phillips is a lead database engineer at ITTIA. He has more than 15 years of software and database development experience. He holds a BS in Computer Science from the University of Washington.

ITTIA
425-462-0046
www.ittia.com