Despite the widespread misconception that some data races are entirely harmless, modern optimizing compilers generate code that can cause incorrect execution when data races exist. Using static and dynamic analysis, programmers can find and eliminate these defects that are often inadvertently introduced into their code.

Programming multicore processors to take advantage of their power means writing multithreaded code. C and C++ were not designed for concurrency, so developers must use a library such as pthreads for those languages. Multithreaded code is more difficult to get right than single-threaded due to the risk posed by entirely new classes of programming defects.

In the rogue’s gallery of concurrency bugs, the race condition is a notorious repeat offender. A race condition occurs when a program checks a resource property and performs an action on the assumption that the property has not changed, even though an external actor has slipped in and changed that property.

A data race is a particular type of race condition that involves concurrent access to memory locations in a multithreaded program. This defect occurs when there are two or more execution threads that access a shared memory location, at least one thread is changing the data at that location, and there is no explicit mechanism for coordinating access. If a data race occurs, it can leave the program in an inconsistent state.

The insidious nature of data races

It is widely assumed that some data races are harmless and can be safely ignored. Unfortunately, this is only true in very rare circumstances. It is best to explain why by introducing an example.

The singleton pattern is a commonplace idiom where the program maintains a reference to a single underlying object and a Boolean variable encodes it if it has been initialized. This pattern is also known as lazy initialization. The following code is an example of the pattern:

if (!initialized) {

object = create();

       initialized = true;

}

… object …

This code is perfectly adequate for a single-threaded program, but it is not thread-safe because it has a data race on the variable named initialized. If called by two different threads, there is a risk that both threads will observe initialized as false at essentially the same time, and both will call create(), thus violating the singleton property.

To make this thread safe, the natural approach is to protect the entire if statement with a lock. However, acquiring and releasing locks can be costly, so programmers try to avoid the expense by using the double-checked locking idiom – a check outside the scope of the lock and another inside. The inner check is there to confirm that the first check still holds after the lock has been acquired:

if (!initialized) {

       lock();

       if (!initialized) {

            object = create();

            initialized = true;

       }

       unlock();

}

  … object …

Superficially, this looks like it will suffice, and indeed, it will as long as the statements are guaranteed to execute in that order. However, an optimizing compiler might generate code that essentially switches the order of object = create() and initialized = true. After all, there is no explicit dependence between those two statements. In that case, if a second thread enters this code any time after the assignment to initialized, that thread would then use the value of object before it has been initialized.

Optimizing compilers are inscrutable beasts. Those that optimize for speed will take many esoteric considerations into account, few of which are obvious to a programmer. It is common for them to generate instructions that are apparently out of order because doing so might result in fewer cache misses, or because fewer instructions are needed.

It is wrong to assume that because the reordering introduced a race condition in the previous example that the compiler is at fault. The compiler is doing exactly what it is allowed to do. The language specification is perfectly clear and unambiguous about this: The compiler is allowed to assume that there are no data races in the program.

In actuality, the specification is somewhat broader: Compilers are allowed to do anything in the presence of undefined behavior. This is sometimes facetiously referred to as catch fire semantics; the specification gives the compiler permission to set a computer on fire if the program has undefined behavior. As well as data races, many traditional bugs such as buffer overruns, dereferences of invalid addresses, and so on constitute undefined behavior. Because compilers are free to do anything, rather than burn the building down they typically do the sensible thing, which is to assume that the undefined behavior will never happen and optimize accordingly.

The consequences of this can sometimes be surprising, even to those who are experts in concurrency and compilers. It can be difficult to convince programmers that code that looks completely correct can be compiled into code that has serious errors.

Another example is worth describing. Suppose there are two threads where one reads a shared variable and the other writes to it. Let’s assume that it does not matter to the reader if it sees the value before or after it has been changed by the writer (this is not an uncommon pattern). If those accesses are not protected by a lock, then there is clearly a data race. Notwithstanding the catch fire rule, however, most programmers would conclude that this is completely benign.

As it turns out, there are at least two plausible ways in which this code could be compiled where the reader would see a value that is wrong. The first way is easy to explain: Suppose the value were a 64-bit quantity on an architecture that can only read 32-bit words. Then both the reader and the writer need two instructions, and an unlucky interleaving might mean that the reader sees the top 32 bits of the old value and the bottom 32 bits of the new, which when combined can be a value that is neither the old nor the new.

The second way in which wrong code could be generated is more subtle. Suppose the reader did the following, where the data race is on the variable named global:

int local = global;  // Take a copy of
                       // the global

if (local == something) {

       …

}

… // Some non-trivial code that does
      // not change global or local

if (local == something) {

       …

}

Here the reader is making a local copy of the racy variable and referring to that value twice. It is reasonable to expect that the value will be the same in both places, but again, the optimizing compiler can generate code where that expectation is unmet. If local is assigned to a register then it will have one value for the purposes of the first comparison, but if the code between the two conditionals is sufficiently nontrivial then that register may get spilled – in other words, reused for a different purpose. In that case, at the second conditional, the value of local will be reloaded from the global variable into the register, by which time the writer might have changed it to a different value.

Programmers should be very skeptical of claims that some data races are acceptable and should strive to find and remove all of them from their code.

Techniques for finding risky defects

When it comes to finding concurrency defects, traditional dynamic testing techniques might be inadequate. A program that passes a test a hundred times is not guaranteed to pass the next time, even with the same inputs and the same environment. Whether these bugs manifest or not is exquisitely sensitive to the timing, and the order in which operations in threads are interleaved is essentially nondeterministic.

New dynamic testing techniques for finding data races are emerging. These techniques work by monitoring the applications as they execute and observing the locks held by each thread, as well as the memory locations being accessed by those threads. If an anomaly is found, then a diagnostic is issued. Other tools help diagnose data races suspected of causing failures. Some companies now offer tools to facilitate diagnosis of data races that allow the replay of events leading up to an anomaly.

Static analysis tools can also be useful for finding data races and other concurrency errors. Whereas dynamic testing tools find defects that occur for particular executions of a program with a fixed set of inputs, static analysis tools check all possible executions and all possible inputs. For performance reasons, tools might place limits on how much exploration is done and thus might not be completely exhaustive; even so, they can cover much more than can ever be feasible with dynamic testing. The advantage of static analysis is that test cases are not required because the program is never actually executed.

Instead, these tools work by creating a model of the program and then exploring the model in various ways to find anomalies. GrammaTech’s CodeSonar finds data races by creating a model that represents the set of locks held by each thread and by performing a symbolic execution of the program that explores execution paths. It records the sets of variables protected by locks and uses this information to find interleavings that can result in shared variables being used without proper synchronization. Similar techniques can be used to find other concurrency defects such as deadlock and lock mismanagement.

Once found, data races are usually easy to fix, albeit doing so correctly can incur a performance penalty. In some cases, there might be a temptation to use the C volatile keyword to correct the data race, but this is not recommended because volatile was not designed to solve concurrency problems, and in any case is a poorly understood construct that is frequently miscompiled. The latest versions of C and C++ have embraced concurrency and support atomic operations. Compiler support for these operations is slowly emerging, and until it becomes readily available, the best approach is to use locks.

To achieve high-quality software for multicore processors, a zero-tolerance policy for data races is recommended. Find them using a combination of both static and dynamic techniques, and take care not to rely too heavily on esoteric compiler techniques to fix them. These defects are so risky and unpredictable that eliminating them systematically is the only safe way to be sure that they do not cause harm.

Paul Anderson is VP of Engineering at GrammaTech.

GrammaTech
607-273-7340
paul@grammatech.com
www.grammatech.com

The ubiquitous nature of embedded software has made source code analysis a critical component of the development process. Gwyn discusses the impact this technology has on software security and reliability, and emphasizes the importance of making static analysis a natural part of a developer’s coding practice.

ECD: As embedded developers turn to multicore processors to optimize performance, how can analysis tools help control inevitable cost and schedule problems?

FISHER: Any new development is an exercise in balancing expectation against risk. In the case of multicore, the naive expectation is always linear acceleration tempered perhaps by some jocular “wouldn’t that be nice” acceptance that the final result won’t be quite that good, but no real understanding of the reality that without significant effort (read: time, money, angst) the result might be slower than the old, interrupt-driven single-core code. So tools have a role to play in terms of helping developers understand the impact of what they’re doing, what pitfalls they’re unwittingly leaving themselves open to, and how to mitigate the associated risks.

Dynamic analysis in this space has received the lion’s share of attention for obvious reasons. If I can see a nice set of graphs over time that shows the performance of my code in action, I can presumably narrow in on problem areas quickly and apply my own knowledge to figuring out what’s going on. The challenge with dynamic analysis is that it depends on a) defining a test set that shows execution problems, and b) the reviewer’s intimate knowledge and understanding of what to do about those problems.

Static analysis, by contrast, assumes little knowledge on behalf of the reviewer and requires no effort to be expended in defining test cases. Every conceivable code path through the application is exercised with as much rigor as every other path. This approach is therefore far more likely to show complex and costly issues such as data races, deadlocks, and resource contention than any constructed test bench. That speaks directly to the bottom line of cost control in what is inevitably a large, over-budget project. Leaving issues such as these in a code base until late in the validation process will cost exponentially more to address than doing so during initial development.

In addition, due to the way that static analysis works by modeling the expected program execution, the finding is accompanied by a detailed walk-through of how the situation is predicted to occur, allowing even a relatively junior resource to interpret the issue at hand, determine whether or not it’s likely to happen, and apply an appropriate design fix. In one example we like to describe in seminars on the subject, a design flaw in a popular open-source database kernel resulted in months of effort expended to identify a deadlock and eventually rewrite key modules to avoid the data race at its heart. This same problem was identified during the first analysis using our tools, which provided a walk-through description enabling developers to easily see that the data race was causing the problem and that the deadlock was merely the symptom.

Contrast a few hours to run the tool to analyze the code and an hour at most to interpret and act upon the result (what turns out to be a one-line fix) with months of community effort to determine an appropriate set of tests, followed by design effort attempting to fix the deadlock, and finally requiring the designer to rewrite the whole thing from scratch.

ECD: How is static analysis introduced in the software development cycle, and how can it be used with existing Integrated Development Environment (IDE) tools?

FISHER: There’s absolutely no doubt that any developer-facing tool that doesn’t integrate seamlessly with any existing tooling is going to face significant friction in deployment. We’ve been selling this message effectively for years, with development managers almost asking the “why wouldn’t you do it this way?” question for us.

Whether developers have migrated to IDEs or their idea of an IDE is a bunch of gVim macros or emacs lisp modules, to them it’s where they live and work. And woe betide any vendor who tries to get them to change. Even if you’re not suggesting that they change tools, and instead asking them to visit somewhere else to see what they might have done wrong in some retrospective manner, your tool is going to suffer waves of disinterest and ultimately become shelfware.

Thus, static analysis has to be part of the developer’s native habitat, and more importantly, it has to work in a way that feels natural and follows the way other tools in that habitat work. For a gVim developer, issuing a ‘:’ command is second nature, so tools in that environment should follow suit. Put that same interaction mechanism in front of a Visual Studio user, and that will make for a fun-filled afternoon of derisive commentary.

At Klocwork we’ve gone through various iterations of technology design, getting closer to the developers themselves. It’s one thing to be resident as a tool within an IDE; it’s another to get the developer to “push the button” to use it. Making a button available is only changing geography, and it does nothing to help with the tool’s fundamental usability.

With this in mind, we’ve recently introduced a new technology that allows static analysis to take place in much the same way as spell checking in a Word document or e-mail. That is, as you’re writing your code and the tool detects a problem with what you’re doing, it can point out the problem in a perceptually instant manner, highlight it with a squiggly underline, and deliver all the value of static analysis with none of the “what do I have to do to use it?” resistance that is typically encountered during any new tool’s introduction (see Figure 1).

(Click graphic to zoom by 1.9x)

Getting a tool into an IDE is tough; getting it to be useful in the part of the IDE the developer truly lives in (the editor component, whatever that looks like in the environment at hand) is really tough. But until you’re there, you’re a distraction.

ECD: Can source code analysis be used to protect embedded devices against potential security threats?

FISHER: Absolutely, source code analysis has a significant role to play in threat identification and validating whatever threat model is being used to determine vulnerability in the device. The connectivity requirement is common in the embedded world today. That connection might be to another chip, or another device, or the whole Internet. In any case, there’s somebody else either sending you information or receiving information you’re sending. That’s the starting point for having to worry about your entire application design.

Static analysis doesn’t typically know, or try to know, anything about your surrounding environment. Tools can sometimes be tuned to perform their analysis within certain data boundaries, for example, such as knowing that a particular input will only range between -20 and +30 because it’s a temperature sensor intended for use in Western Europe. But that kind of thing is discouraged because you’re allowing the user to define limits to what the modeling technology naturally does – that is, assume nothing and point out everything that looks wrong.

In the case of threat detection, we’re most worried about how the data you’re interacting with from the outside world is used internally. Is it used to create a buffer into which you’ll read data (code or value injection), or is it used for memory allocation (Denial of Service or DoS), or perhaps interpreted as a reference into an internal data structure (hijacking or redirection)? This kind of data and path validation – that is, the path that tainted data follows from the outside world to its point of use within the code – is natural for source code analysis, as it accomplishes this modeling in order to perform everything else it does.

As a wonderful gentleman at a defense contractor once said to me, “Son, it’s a bomb; it’s supposed to blow up. We just want to be sure it doesn’t get hijacked on its way.”

ECD: What educational events or online classes does Klocwork offer to help embedded designers get started with its code analysis tools?

FISHER: Like any commercial organization attempting to encourage users to gain value from its tools, Klocwork provides a full suite of educational and professional services, running the gamut from introductory materials aimed at first-time users, to more advanced courses targeting secure coding and threat modeling, to full-on deployment services and mentoring.

We also recently introduced the Klocwork Developer Network (http://developer.klocwork.com), a website serving our users and acting as a repository for online courses, video tutorials, in-depth courseware, and the usual variety of community forums and ticketing.

Each customer has something unique they wish to gain from a tool such as source code analysis, so a large part of our educational focus is on internal champions, people who take knowledge of how the tools work and how other organizations have applied them and leverage those lessons in deploying our tools for their own use. A large part of that is learning from the community, so I’ve been thrilled to see the fast uptake that the Klocwork Developer Network has seen amongst users, both as a less formal mechanism for interacting with our staff, but most importantly as a way to learn from other customers.

(Click graphic to zoom)

Gwyn Fisher is CTO of Klocwork.

Klocwork info@klocwork.com www.klocwork.com/blog www.facebook.com/klocwork www.twitter.com/@klocwork www.klocwork.com

You’ve been diligent and used static code analysis to identify defects early during development. Great. So now what? Finding defects is just the first step in the process of ensuring software integrity. Contextual information on every identified defect is essential to prioritize fixes and maintain bug-free software.

Embedded software is ubiquitous and provides critical functionality in a variety of devices, from the latest smartphones and gaming gadgets to life-saving medical devices. Engineering organizations creating embedded software understand that ensuring quality of code is a key differentiator and competitive advantage. Along with other methods of testing and verification, many companies have taken advantage of the benefits of code testing with modern static analysis to identify defects early in development. During the past few years, various reports by embedded market research firm VDC Research indicate strong growth in companies adopting static analysis as a critical test automation tool. Modern static analysis is arguably the most cost-effective, automated, and repeatable way to meet the challenge of ensuring the quality of complex software.

A strong reason driving this growth is that the technology used to identify critical defects such as memory corruptions, resource leaks, null pointer dereferences, and invalid memory accesses has matured to a point where finding large numbers of hard-to-find defects that traverse function and file boundaries can now be accomplished accurately, resulting in a very small number of false positives. However, the real innovation lies in providing contextual information for every defect identified. A developer needs to know why the defect exists, what impact it will have, and where it needs to be fixed.

The answer to the question of where it needs to be fixed is not as simple as knowing the file name and line number. Code branching and merging for versioning, reuse of code, and reuse of code components for development productivity allow a defect to make its way into multiple versions and products.

Consider the case of a software team with multiple branches for various versions of the product. A bug in one of these branches might exist in one or more other branches due to code replication. In another case, consider a team creating the framework to support applications for smartphones. Because they might be porting the framework onto various platforms like Windows, Android, or iPhone, it is critical that the static analysis results clearly indicate whether identified defects exist in just one place or on multiple platforms. Similarly, when software is created by aggregating from multiple sources, it is a nightmare when a particular component is used in various products, as a defect in one third-party component might end up affecting all the different products that include it.

Multiple branches for different versions of an operating system

Imagine a software development team responsible for creating a new Operating System (OS) for mobile smartphones. Because multiple mobile phone vendors (OEMs) must be supported, every vendor in the source control management system needs a development branch. In addition, each vendor typically has multiple branches for different releases and product generations. The picture starts to get complex very quickly.

Static analysis performed on every branch of the code produces a list of defects. However, depending on when a defect is introduced, it could exist in all versions or a subset. When looking at a single defect in isolation in a single branch, the challenge for developers is that they can’t gauge the severity of the defect without knowing where else it is present. A defect that is not limited to a single version or one OEM client would be severe, and fixing it would need to be prioritized over anything else. Additionally, a developer writing code to fix the defect needs to know exactly which branches in the source control management system the fix needs to be checked in. Analysis results that pinpoint the exact location where the defect exists and provide information such as the branches where defects occur are highly valuable to developers (see Figure 1).

(Click graphic to zoom by 1.9x)

A single framework for multiple platforms

On the flip side of branching, there is often a need to write code designed to run on multiple platforms. A software component such as a framework for mobile applications is usually built to run on various types of mobile phone platforms. For embedded devices, a common requirement is to build 32- and 64-bit versions of the same code base. Let’s take a simple example:

gcc –m32 -c foo.c

// 32-bit compile. Contains a null pointer dereference defect.

gcc -c foo.c

// 64-bit compile. Contains the same null pointer dereference defect.

A defect in foo.c that gets triggered in both 32- and 64-bit binaries will be detected and reported as a single defect. However, because the source code is the same, a sophisticated analysis would not report it as a duplicate defect. Duplicates are as harmful as false positives in losing the developer’s trust in the static analysis solution.

Sharing common code components

In this final example, consider a team developing the platform software for a family of networking switches. Because the functionality provided by the platform software must be implemented in all products, this code component will be shared (see Figure 2). For developers working on this team, the best assessment of the severity of a defect reported by static analysis is not only the impact it will have on one switch product, but also information on all the products that use this platform software component.

(Click graphic to zoom by 1.9x)

A product is usually created by combining many such shared components. Each component is not only a project itself, but also a part of various other projects using it. The analysis result needs to identify that a defect in this shared component has an impact on the various projects using it.

Taking the guesswork out of code testing

The adoption of modern developer-side testing methods such as static analysis is a positive trend in the embedded software industry. The technology has matured to a point where it is a strong weapon in the software engineer’s arsenal. Without needing to create elaborate test cases and testing infrastructures, static analysis automatically finds critical defects as code is written and compiled. However, for static analysis to become a developer’s most valuable tool, the analysis must provide answers to questions such as “What is the impact of this defect?” and “Where do I need to check in the fix?” to help prioritize fixing the identified defects and take the guesswork out of ensuring that software is as bug-free as possible.

(Click graphic to zoom)

Rutul Dave is senior development manager at Coverity. He has several years of software development experience in embedded and real-time systems, including work developing bleeding-edge technology systems at Procket Networks, Topspin Communications, and Cisco Systems. When not evangelizing about the benefits of software integrity, Rutul scratches the coding itch by developing mobile apps and understanding the Linux kernel. He received his Master’s in Computer Science with a focus on networking and communications systems from the University of Southern California.

Coverity info@coverity.com www.coverity.com

Due to the rise of multicore processors, programmers increasingly need to write multithreaded code. Pitfalls such as race conditions, starvation, and deadlock make it hard to get such code right, and traditional testing can only help so much. New approaches based on static analysis are proving effective at quickly finding difficult bugs.

Although decades of advances in miniaturization have yielded enormous performance gains for single processors, it appears this era is coming to a close. The best bet for achieving significant additional performance with single chips is through multiple cores, but only if software can be programmed to take advantage of them.

Unfortunately, concurrent programming is difficult. Even expert-level programmers familiar only with single-threaded programming often fail to appreciate that concurrent programs are susceptible to entirely new classes of defects such as race conditions, deadlocks, and starvation. It is difficult for humans to reason about concurrent programs, and some aspects of programming languages themselves are ill-suited to concurrency. Consequently, experts frequently stumble over these hazards. The following discussion describes common concurrency pitfalls and explains how static analysis tools can find such defects without executing the program.

Consequences of race conditions

A race condition arises when multiple threads of execution access a shared piece of data and at least one of them changes the value of that data without an explicit synchronization operation to separate the accesses. Depending on the interleaving of the two threads, the system can be left in an inconsistent state.

Race conditions are especially insidious because they can lurk undetected indefinitely, and only show up in rare circumstances exhibiting mysterious symptoms that are difficult to diagnose and reproduce. In particular, they are likely to survive through testing into deployed software. At best, this means increased development times; at worst, the consequences can be devastating.

One reason the Northeast Blackout of 2003 was so widespread was that a race condition in a computerized energy management system caused misleading information to be communicated to the operators. As Kevin Poulsen noted in a 2004 article (www.securityfocus.com/news/8412), “the bug had a window of opportunity measured in milliseconds.” The chances of a problem like this manifesting during testing are infinitesimal. In another case, a race condition in iOS 4.0 through 4.1 (now fixed) meant that any person with physical access to an iPhone 3G or later could bypass its passcode lock under certain conditions.

An example of a simple race condition is shown in Figure 1. A manufacturing assembly line with entry and exit sensors maintains a running count of the items currently on the line. This count is incremented every time an item enters the line and decremented every time an item reaches the end of the line and exits. If an item enters the line at the same time that another item exits, the count should be incremented and then decremented (or vice versa) for a net change of zero. However, normal increment and decrement are not atomic operations; they are composed of a sequence of individual instructions that first loads the value from memory, then modifies it locally, and finally stores it back in memory. If the updating transactions are processed in a multithreaded system without sufficient safeguards, a race condition can arise because the sensors read and write a shared piece of data: the count. The interleaving in Figure 1 results in an incorrect count of 69. There are also interleavings that can result in an incorrect count of 71, as well as some that correctly result in a count of 70.

(Click graphic to zoom by 1.8x)

For this example and for race condition bugs in general, standard debugging techniques may be ineffective for several reasons.

Rare occurrence means a reduced chance of noticing there is a problem. If the problem manifests infrequently, it may never show up during testing. The issue is twofold. Firstly, the number of possible interleavings of the instructions in two threads can be huge and increases enormously as the number of instructions grows. This phenomenon is known as combinatorial explosion. If thread A executes M instructions and thread B executes N;instructions, the possible interleavings of the two threads are:

(Click graphic to zoom by 4.8x)

For example, given two trivial threads with 10 instructions each, there are 184,756 possible interleavings of the instructions. Real-world software is large and complex; testing every interleaving is simply impossible. Secondly, even if testers can identify a few interleavings that merit inspection, it is difficult to set up test cases to ensure they actually occur because thread scheduling can be highly nondeterministic.

If exhaustive testing is intractable, then what can developers do? One extremely useful approach is static analysis. Advanced static analysis tools such as CodeSonar use highly sophisticated symbolic execution techniques to consider many possible execution paths and interleavings at once. These techniques can find race conditions and other concurrency errors without needing to execute the program.

Several factors make race condition diagnosis difficult. Firstly, the symptoms can be perplexing. In the Figure 1 example, the running count will usually be correct, but sometimes too high and other times too low. Secondly, programmers unaccustomed to considering the particular pitfalls of multithreaded programming may spend a lot of time puzzling over the code before the possibility of a race condition occurs to them. Advanced static analysis tools are especially helpful in this regard. They identify race conditions by examining patterns of access to shared memory locations; that is, they focus on the race itself, not its symptoms. When a race condition is identified, an advanced static analysis tool will report it along with supporting information to aid the user in evaluation and debugging. The onus on the programmer is substantially reduced.

More complexity, more bugs

Race conditions are typically avoided by using locks to protect shared resources. However, locks can introduce performance bottlenecks that might prevent the program from taking advantage of the full potential of multiple cores, so programmers must exercise care in using them. It can be tricky to write code that uses locks effectively, and this complexity can lead to a different set of problems, namely deadlock and starvation.

In a deadlock, two or more threads prevent each other from making progress because each holds a lock needed by another. Figure 2 shows how a deadlock can arise with two locks used to protect two shared variables. In this example, multiple assembly lines share a count of the total number of items currently under assembly, and a second bad_items value records how many finished items failed quality control. One thread acquires the lock on count, another acquires the lock on bad_items. Neither thread can obtain the second lock it needs; thus neither can carry out its operations, nor can it get to the point where it will release its lock. As neither update can be completed, both threads are completely stuck.

(Click graphic to zoom by 1.9x)

Static analysis tools can identify software at risk of deadlock by flagging situations where the same locks can be acquired in different orders by different threads, such as the threads shown in Figure 2. Eliminating all such cases is sufficient to ensure that the system cannot become deadlocked.

Starvation is another problem that occurs in multithreaded programs that use locks. A thread can starve if it is waiting for a resource currently held by another thread that takes a very long time. For example, suppose the aforementioned manufacturing automation system includes a regular audit thread that examines all entry and exit records to ensure that the running count matches total items entering less total items exiting. The audit thread would need to hold locks on the count and on all sensors, so all updates must wait for the audit to finish. If the audit runs for a long time, updates can be significantly delayed. If it runs too long, the next audit may manage to acquire all the locks and start running before the outstanding thread can make any progress. In the worst case, some or all of the updates may never have the opportunity to run.

Static analysis can provide significant value here by posing questions such as, “Is there a call to a long-running library function while a lock is held?” Tools such as CodeSonar also provide mechanisms for users to add their own checks. If an in-house function f() is known to have a long running time, engineers could add a custom check that triggers a warning whenever f() is called by a thread that holds one or more locks.

Multithreading adds entirely new classes of potential bugs to those that embedded developers must consider, making it significantly more difficult to find bugs of all kinds. The latest generation of static analysis tools can help with both of these issues.

(Click graphic to zoom)

Paul Anderson is VP of engineering at GrammaTech, where he manages the engineering team and architects static analysis tools. He has worked in the software industry for 20 years, helping organizations including NASA, the FDA, the FAA, MITRE, Draper Laboratory, GE, Lockheed Martin, and Boeing apply automated code analysis to critical projects. He received his BSc from King’s College London and his PhD in Computer Science from City University London.

GrammaTech paul@grammatech.com www.grammatech.com

With modern military systems increasingly relying on software, new techniques are being adopted to decrease costs and increase the chances for mission success. As a result, static analysis has been gaining traction in the software development community based on its deep analysis capabilities before runtime. These static analysis tools – which augment but do not replace traditional test and debug methods – find integration errors long before integration occurs, eliminating costly late-stage integration issues.

As military equipment and vehicles become increasingly modernized, they inevitably become more technologically complicated as well. In many instances, these machines represent a delicate blend of hardware and software, both of which must interact perfectly. Since the software components of these machines must work as reliably as possible in the field, it is important to perform extensive debugging and testing using a combination of techniques to eliminate defects before they cause extended delays or overruns. This is often impractical since there is usually no way to reliably test code until it is actually implemented into the equipment on which it will eventually run. Since software seldom works perfectly the first time it is executed, going back and fixing errors with conventional development methods and tools hinders productivity and diverts valuable resources and personnel from other projects that need to be completed.

For instance, the Boeing 787 Dreamliner was delayed for two years because of a combination of hardware and software defects. Often, such problems are intertwined; in the case of the Dreamliner, one particular delay was the result of defects in the software that controlled the braking system. It is important to note that conventional testing did not reveal this defect before it actually became a problem and caused costly setbacks and other complications in the development process. Modern techniques like static analysis can increase the probability of problems like memory corruption and user-after-free being discovered sooner, as well as assist in achieving DO-178B’s Design Assurance Levels (DALs). A more efficient and cost-effective path than the traditional V-model, static analysis works in concert with traditional test and debug methods to ease integration woes and expense.

Static analysis and the development process

Software development tends to follow a specific life cycle. One example is the V-model (Figure 1), which is commonly used in aerospace engineering. The V-model represents a holistic approach to development that attempts to reconcile project definition and the testing process over a period of time. The model begins with establishing the scope of a project, including its concepts of operation, its requirements and architecture, and the specific details of its design.

(click graphic to zoom by 1.2x)

This process starts out very abstract at higher levels and becomes gradually refined and detailed over the course of the design process. As the design is implemented, it becomes more expensive to fix problems later in the development cycle. Once the project is mostly complete and is being tested, going back and making repairs to its fundamental aspects becomes prohibitively expensive.

Ultimately, many software problems are caused and then worsened by an inefficient development model and imprecise debugging procedures. Development efficiency and cost-effectiveness can be improved by eliminating software defects as early as possible with modern techniques like static analysis.

Specifically, static analysis is a technique for finding defects in software without running it. It works by examining product source code, starting from individual functions, working its way up to modules, then ultimately the entire program. Static analysis can find many different kinds of defects, including memory errors in C/C++ programs. For example, static analysis can detect an error in this simple code fragment:

int a[10];

for(int i = 0; i < 10; i++); {

a[i] = 0;

}

Sometimes it is hard for human beings to see problems in software code because they see what they want to see instead of what is actually there. When this code is turned into an executable program by a compiler, the compiler reads the source code in a mechanical and precise way, ignoring human cues like indentation and spacing. A compiler will read the example in the following way:

int a[10];

for(int i = 0; i < 10; i++)

;

{

a[i] = 0;

}

Once the code has been reformatted as shown, an extra “;” character that causes the program to have a completely different meaning suddenly becomes much more visible. If this error remains uncorrected in the final program, the array access a[i] will execute only once when i = 10. The result is an assignment to a memory location past the end of the array, which might cause the program to crash.

Static analysis examines the code as mechanically and precisely as a compiler would. However, instead of blindly translating the code into an executable program, a static analyzer looks for paths where the code’s function diverges from how the developer originally intended it to work. Static analysis can be done by techniques as simple as pattern matching or as advanced as interprocedural dataflow analysis and Boolean satisfiability. Regardless of technique, static analysis is custom-built to look for cases that human developers are likely to overlook or get wrong. This presents a far more efficient alternative to comprehensive line-by-line code auditing, which is not cost-effective for larger software systems.

Combine static analysis and traditional debugging

Compared with a traditional method such as functional testing, static analysis presents a different set of trade-offs. Traditional testing can only detect errors in code that is actually tested, whereas static analysis can find defects in all code without any tests. Sometimes, errors found through testing are hard to reproduce and pinpoint to a specific problem in the source code. Static analysis can find problems in a repeatable, predictable fashion and will always point to a specific location in the code. On the other hand, traditional testing can find functionality errors that static analysis cannot, because static analysis does not attempt to compare the behavior of the program against an expected result. Static analysis can also lose precision when analyzing deep program properties, so it might miss some defects. Therefore, static analysis is meant to augment the effectiveness of traditional methods instead of replacing them outright.

Since static analysis works with existing toolsets and compilers, there is no need to change current development practices. Static analysis can be applied frequently (even on nightly builds) for the duration of the project starting immediately after the first line of code has been written. Essentially, if the project’s code base can be successfully compiled, static analysis can be used to debug it and eliminate problems long before the project is delivered to quality assurance personnel for final testing.

Integration benefits of static analysis

Static analysis also works well in a team-oriented environment. Developers can use static analysis to check each other’s contributions for consistency and check for conflicts caused by code written by different team members. This can work within a single project or even scale to larger scenarios where integration between several projects is required to create a complicated system.

Static analysis can also do more than find simple code defects; as mentioned, it is capable of analyzing the interactions between different components before they are integrated together. In the V-model of software development, testing and verification start with individual components. When these components meet their low-level specifications, they are integrated together for higher-level testing against system requirements. If software integration is performed strictly according to this ideal, the integration phase introduces a high degree of risk because the individual components will be interacting for the first time. These interactions are likely to expose problems with the original project specifications, particularly regarding how the system requirements and architecture are translated into detailed design requirements and source code. Static analysis tools can analyze the interactions between software components as soon as they are written, catching some of these expensive integration problems during the implementation phase. The tools accomplish this by finding problems in Application Programming Interface (API) usage by analyzing code across procedure boundaries even if the procedures are in different software components.

Maximizing efficiency with static analysis

Software is being used to improve military systems such as smart bombs and unmanned drones in ways that were once impossible. Static analysis tools, such as those offered by Coverity, are also a part of this trend. Static analysis tools – when combined with traditional test and debug methods – effectively analyze software and detect code errors before runtime, minimizing risks and costs while maximizing the value of investment in software development.

Andy Chou is chief scientist and cofounder of Coverity. He is responsible for advancing source code analysis technology as well as furthering the state-of-the-art in software quality and security industry-wide. He received his Ph.D. in Computer Science from Stanford University and his B.S. in Electrical Engineering from UC Berkeley. He can be contacted at achou@coverity.com.

Coverity 415-321-5200 www.coverity.com

Static analysis tools are becoming more integrated into the software development process. Saving data from the compiler, change history, and error information during the process instead of as a post-code step can make static analysis more productive.

Advanced static analysis tools are becoming increasingly critical in embedded systems development. Going well beyond older static analysis tools that were, in effect, coding style checkers, new tools statically analyze a source program’s control and data flow, thereby detecting bugs and vulnerabilities such as potential buffer overflows, uses of uninitialized variables, accesses through null pointers, and susceptibility to security attacks (SQL injection, cross-site scripting, and so on).

However, these advanced tools raise several issues. First, the tools need to understand the semantics of the program being analyzed – that is, they have to compile the program – to perform the required control- and data-flow analysis. To do this, they must be closely integrated into the build environment, so that all include files or other specification modules that might be needed at compile time are identified and available. Second, the amount of output produced by these advanced tools can be daunting, with each diagnostic message requiring careful scrutiny to determine if it reflects a real problem, and if so how, to address it.

Integrating the static analysis tool more closely with the software development tool chain can alleviate both of these challenges. In the first case, integrating the static analysis tool tightly with the compiler largely eliminates build environment problems and makes the user interface simple and familiar. In the second case, the extensive output can be managed by storing all output in an historical database, allowing the programmer to focus on deltas between a known good release and the current state of the source, rather than dealing with all the messages at each step.

Integrating with the compiler

A static analysis tool that goes beyond simple syntax checking generally needs much of the power of a compiler front end so that it can base its analysis on the semantics of the program. This is because the same syntactic form can often have different interpretations based on the meanings of its constituents. For example, the expression F(N) in Ada might be (among other things) an array reference, function invocation, or type conversion.

Having access to the underlying semantics of the program allows the tool to follow every name appearing within the program back to the declaration that introduces that name, even in the presence of overloading, generic templates, or renaming. The tool will know the type of every object and every expression, and will identify where any implicit runtime checks occur. These implicit runtime checks might include a check for dereferencing a null pointer and a check for indexing outside the bounds of an array. Even languages without implicit runtime checks can define certain runtime actions to have unspecified semantics, such as an integer arithmetic overflow or an array index that is out of bounds. A static analysis tool will need to know when the language semantics allow such unspecified (and thus unpredictable) behavior.

Because of the need to include the power of a compiler front end, many static analysis tools are built on top of preexisting compiler technology for the language of interest. Unfortunately, the compiler technology chosen by the tool’s builder might be unrelated to the compiler used by the tool’s customer. When this happens, the static analysis tool might not work on the customer’s code as written.

For example, if the customer program uses compiler-specific features (such as interrupt handling or special memory-mapping facilities), then there is no guarantee that they will be supported at all, or in the same way, by the static analysis tool’s underlying compiler front end technology. Even for portable code, the customer’s compiler and the static analysis tool’s underlying technology might have different bugs or subtly different interpretations of the rules of the language. And even when the interpretations match, the commands to compile the program – the command line switches that control source code search paths, preprocessor support, and other features – might differ significantly. Thus, the build process for a complex program can be difficult to translate into a make process for performing static analysis on the program.

To address these issues, the clear solution is to integrate the advanced static analysis engine with the same compiler technology the customer is using. The static analysis engine must therefore be somewhat independent of the intermediate representation used by any particular compiler technology, so that the tool can be easily adapted to support multiple compiler front ends.

One approach is for the static analysis engine to have its own intermediate representation specifically designed to support the advanced analyses the tool performs. Adapting to support a new compiler front end requires writing a translation module that transforms the compiler’s intermediate representation (the output of the front end) to the program representation used by the static analysis engine. The translation module outputs the result into a file for later use. The intermediate language translator can either be linked to the compiler front end or run as a stand-alone program, reading the compiler’s intermediate representation, transforming it, and then writing out the analysis engine’s intermediate representation. This process is illustrated in Figure 1.

(Click graphic to zoom by 1.6x)

When this integration approach is adopted, static analysis becomes just another part of the build process, performed either during compilation or, to take advantage of whole program analysis, during the link step. A key advantage to users is that the invocation of the static analysis tool only involves providing an additional command line switch to the compiler and/or linker. There is no need to create a specialized build script for the tool or maintain two sets of sources (one that works with the compiler, and one that works with the static analysis tool).

Integrating with the development environment

Because software development is often conducted through graphical Integrated Development Environments (IDEs) such as Eclipse, it is natural to integrate the static analysis tool and the compiler into the IDE. The overall interface to the tool will then be immediately familiar to the programmer, reducing the learning curve and increasing the likelihood that the tool will be used on a regular basis.

Messages generated by the static analysis tool must be handled like error or warning messages generated by the compiler, and managed and viewed in the same way by the user. Given that multiple IDEs are in use, each with its own message format, the static analysis tool will need to represent its messages so that they can be readily transformed into whatever format the IDE expects.

A natural choice for message representation is XML, given its tagged, self-describing approach to capturing message characteristics. A side benefit of using XML is that it helps simplify the process of internationalizing the application, so that the messages can be displayed in the natural language preferred by the customer.

Integrating with an historical database

Once the advanced static analysis tool is integrated with the compiler and IDE, the next issue is dealing with the potentially large number of messages that such a tool can provide. Because advanced static analysis tools are looking for possible runtime logic errors and security vulnerabilities, they have to simulate the runtime program execution (identify the set of potential execution states) and determine under what conditions an undesirable state might be reached. Unfortunately, this is rarely a simple case of yes or no. There are many shades of grey where the level of vulnerability depends on factors that might be unknown to the tool or beyond its powers of analysis.

This issue is sometimes phrased in terms of soundness versus precision. A tool searching for problematic constructs is said to be sound if it identifies all of the problems it is looking for (no false negatives). But soundness generally comes at the expense of precision. The tool could generate a large number of false positives, which are warnings or errors identifying issues that are not real problems. Consider this simple example using C-like syntax:

int k, m, n;

… // Complicated code that assigns a positive value to m

… // and that does not assign to n

if (m<0){

   k=n;

   …

}

A tool might not be able to deduce that the m<0 condition on the if statement is false, and thus might warn that the body of the if statement is referencing an uninitialized variable (n). The actual problem is the opposite: The body of the if statement is code that will never be executed, sometimes referred to as dead or unreachable code.

A tool developer must decide whether to opt for soundness (make sure that no actual violations go undetected) or precision (make sure that all reported violations are real errors). When a tool is intended for safety-critical or high-security systems, the scales are tipped to soundness. The developer using such a tool must have confidence that all violations are detected. But this raises the issue mentioned earlier regarding how to deal with the potentially large number of false positives that could be generated. This problem is especially noticeable when the tool is applied to legacy software (code that was developed before applying the static analysis tool). The number of messages a user needs to review for a large application can be daunting.

Integrating the advanced static analysis tool with an historical database makes productive use of the tool, minimizing the problems caused by false positives, even for a complex application developed prior to the tool’s use. The critical concept is the notion of a baseline and the ability for the tool to highlight the deltas relative to such a baseline. By recording all the results of each tool run in an historical database, the tool can identify deltas (changes) between any two runs.

Data becomes more useful

For the comparison between analysis runs to work effectively, messages must be uniquely identifiable without referring to specific line numbers, which can switch from one version of the source code to another without any significant change. One way to identify a message without a line number is to record the text of the message (or the corresponding XML), along with the name of the function in which it appears, and a sequence number if the text of the message is identical to some prior message in the same function.

Presuming messages are stored in the database using this line-number-independent unique identifier as the key, the tool can then easily identify whether a given message is new or has been generated previously. This keeps the overall size of the historical database manageable. Rather than repeatedly storing the text of all messages for all invocations of the tool, the tool only needs to store the text of a given message once, along with an indication of the range of tool invocations where the message appeared (which run was the first where the message was generated, and which run was the first where it did not appear).

This historical database makes it straightforward for the user interface to display or highlight only those messages that are new since a specified baseline. This allows the tool to be used effectively even on large applications with significant amounts of legacy code. A known good release of the application can be run through the analysis tool as a baseline. The current development version of the application can be analyzed, with the results from analyzing this known good release as the baseline. Those working on the development version can focus on any messages associated with changes they have made since the known good release, rather than having to wade through messages that relate to legacy code. Eventually, effort can be devoted to going through this backlog of messages, but that can be scheduled at a time that is convenient or corresponds to a larger reengineering effort.

Another benefit provided by integration with an historical database is the ability to collect comments from users who review the analysis results. In some cases, a particular message might require significant investigation to understand the possible implications. It is important that this work be captured. The historical database is a natural place to record what the user learns.

In addition, if the user determines that the identified code is safe and secure, the historical database can record that the given message should be suppressed from subsequent output, and can record the supporting rationale for suppressing the message. Alternatively, if the identified code needs to be changed, the historical database can record the Program Trouble Report (PTR) ID assigned to the problem, allowing traceability between a problem-tracking system and the analysis tool’s historical results. When the tool detects that a message with an associated PTR ID has disappeared, it can be configured to directly notify the problem-tracking system that the associated PTR record can be closed. Automating the process of closing PTRs can provide significant relief to a typically overburdened quality assurance team.

Static analysis as a key component of the development process

With applications getting larger and more complex, advanced static analysis tools play a key role in modern software development by significantly reducing the effort needed to find bugs and vulnerabilities that can compromise a system’s reliability, safety, or security. But many organizations are not yet taking full advantage of these tools, often because of the potentially high entry barrier to incorporate them into the daily software development process (builds, regression tests, and other steps).

As discussed previously, two important steps can reduce this entry barrier: tool integration with the compiler technology and with an historical database. This is not simply a theoretical proposal. CodePeer, an advanced static analysis tool developed jointly by SofCheck and AdaCore, serves as an automated code reviewer for Ada. This tool has been fully integrated into AdaCore’s GNAT Pro Ada development environment and is invokable through the GNAT Programming Studio IDE.

Integration with the compiler largely eliminates the challenge of porting the source code to the analysis tool. The same compiler front end that successfully compiles the source code can also generate the intermediate representation that the advanced static analysis engine needs for more in-depth analyses. Furthermore, the same command line switches, source code structure, and make files can be used to compile and statically analyze the code. The compiler front end will automatically handle any implementation-specific features used by the application.

The second major step toward reducing the entry barrier is integration with an historical database, which allows developers working on large systems to focus on their recent changes and defer reviewing issues in previously released legacy code until a more appropriate time. Additionally, integration with the database allows developers to record the results of reviewing the tool output and the rationale behind decisions to either suppress the message or file it as a PTR. Finally, the database automatically verifies a fix and closes the PTR. With these two steps, static analysis can become an important and productive tool in the embedded software developer’s toolbox.

S. Tucker Taft is founder and CTO of SofCheck, Inc., based in Burlington, Massachusetts. Tucker founded SofCheck in 2002 to provide tools and technology for enhancing software development quality and productivity. Prior to that, he was a Chief Scientist at Intermetrics, Inc., where he led the Ada 95 language revision effort. Tucker holds a BA in Chemistry from Harvard College.

SofCheck
781-750-8068
info@sofcheck.com
www.sofcheck.com