Moving target: EEMBC evolves its benchmark suites to keep pace with the multicore revolution - Q&A with Markus Levy, Founder and President of EEMBC

1Constantly evolving, the EEMBC’s focus has shifted from benchmarking microprocessors to SoCs and even entire systems, targeting the automotive, networking, and smartphone/tablets markets. Additionally, the emergence and constant expansion of the multicore processor have brought added challenges as EEMBC benchmarks such real-world, yet ever shifting, multicore targets.

Remind us briefly why the EEMBC was started, by whom, what its mission is, and what it provides to which industries.

LEVY: I started EEMBC in 1997 (with some guidance by guys such as Derek Meyer and Geoff Lees) because we realized that there was a clear need in the embedded industry for standardized benchmarks that all processor vendors could utilize. At the time, there were de facto standard benchmarks such as the infamous Dhrystone MIPS, but more often than not, processor vendors used their own proprietary benchmarks, which, of course, meant that there was no way to make apples-to-apples comparisons.

EEMBC has evolved considerably since its inception, with benchmarks going from a microprocessor focus to an SoC focus and even benchmarking the entire system. However, the mission has remained the same – to provide real-world benchmarks to help designers select the right embedded processors for their systems and to ensure that the benchmarking methodology will provide fair and reasonable comparisons. The EEMBC benchmarks are targeted at a variety of industries including automotive, networking, and smartphones/tablets.

Since multicore processors’ emergence onto the embedded scene in recent years, what are the top 3 technical challenges faced by the embedded computing industry and the EEMBC?

LEVY: The top three technical challenges are software, software, and software. Seriously, with the emergence and continued expansion of multicore processors, system developers are challenged with either converting their legacy applications to take advantage of the parallel processing offered by multicore or by understanding how to create new applications that can utilize some of the extremely complex hardware accelerators included in many newer-generation SoCs. For example, even if a legacy application utilizes a threaded programming model, moving to multicore entails dealing with issues such as data races. Alternatively, writing an application from scratch to deal with complex hardware accelerators is a daunting task, and programmers are often unwilling to take this risk, especially considering that the SoC functions could change on the release of the next-generation device.

As this topic relates to EEMBC, you have to look back a decade ago when processor benchmarks were entirely focused on single-core devices. But over time, as the processor evolved into the multicore SoC, benchmarks have also had to evolve in order to adequately measure these application-specific and complex devices. In other words, you’re not able to judge an SoC’s capabilities just by running a standard benchmark on the processor core (although EEMBC’s CoreMark has certainly grown in popularity). Hence, the benchmarks have evolved into system tests, such as BrowsingBench, which tests the page-load performance of a complete mobile phone. Likewise, EEMBC’s networking benchmarks have evolved into DPIBench, which tests the sustained throughput of full-fledged routers while performing deep packet inspection searching for virus-infected files.

While BrowsingBench and DPIBench are system-level benchmarks, they are implicitly testing the underlying multicore hardware (as well as software stacks). Take BrowsingBench as an example. The primary function of this benchmark is to perform html page loads on a client device (i.e., smartphone, tablet). The page-load operation involves a series of steps that includes serial functions (i.e., enter or click URL, fetch initial HTML, parse the HTML, and determine the workload) and parallel functions that could take advantage of a multicore device (i.e., parsing, Javascript, image decoding, page rendering after all elements have been assembled, and animation). Inherently, if the client device’s browser and operating system are designed appropriately, we will see significant performance improvements on these parallel functions.

How are these technical problems best solved, by industry and the EEMBC?

LEVY: EEMBC is solving these problems by continually evolving its benchmark suites to take advantage of (and stress) multicore devices, and at the same time the benchmarks are becoming much more real-world in nature, because that’s the only way to accurately reflect the device’s capabilities. However, the industry as a whole needs to continually strive for better development tools that will help analyze and automate the application’s ability to take advantage of the multicore hardware.

Other industry organizations, such as the Multicore Association, are putting together Application Programming Interfaces (APIs) that will make it easier for programmers to develop portable and scalable applications. These APIs will also allow semiconductor vendors to enable their customers to take advantage of the (sometimes) extremely complex hardware accelerators, many of which don’t even get used today because customers are unwilling or unable to make the investment to read a 1,000-page manual for programming these accelerators. PolyCore Software is an example of one company that has already commercialized tools based on these APIs; these tools allow a system developer to easily assign software functions to specific cores (or nodes) within the system. But more importantly, the tools make it much easier to move those functions to different cores to take advantage of a changing SoC topography.

Over the next 5 years, what characteristics and features will contribute most to multicore processor performance gains? Will it be enough to keep up with embedded industry demands?

LEVY: Certainly, vendors will continue to add more cores and accelerators to increase performance, and hopefully they will also provide the tools to support this. Another hardware-related area that will be beneficial will be increasing memory throughput and higher bandwidth – somehow you have to feed these hungry multicore processors. Software tools will also continue to improve: A tremendous amount of effort is going into this area from a multitude of vendors such as CriticalBlue and PolyCore Software. Although the silver bullet for multicore programming will never arrive, programmers are learning to work with what’s available and are changing their expectations, realizing that the move to multicore will require a fair amount of effort.

Which types of multicore processors have the market edge now? Why? What about 2 years from now? 5 years from now and beyond?

LEVY: The types of processors with the market edge today are entirely dependent on the market that they play in. For example, in some areas, high-performance SMP devices are best utilized, especially for applications that have traditionally used programming frameworks such as Pthreads.

However, inevitably, the types of processors that will succeed in the future will be the SoCs that provide hardware-accelerated functions. It’s the only way that applications will be able to meet their performance-power budgets. In other words, with homogeneous SMP devices, the performance gained by increased core count is not scalable. For example, the more cores that share a common bus structure, the more that each core must compete for memory bandwidth. This problem can be alleviated by designing chips that divide cores into clusters, where each cluster can operate autonomously if necessary.

What plans does the EEMBC have to expand its offerings in the future, and how can the industry get involved?

LEVY: As I mentioned earlier, EEMBC has already begun expanding its offerings to address the system-level performance measurement. I see this trend continuing (especially in the mobile, automotive, and networking areas), although there will always be a need for C-coded benchmarks that purely address the CPU + memory subsystem. As an industry association, EEMBC always encourages companies to get involved with the definition and development of new benchmarks. Our philosophy is “the more the merrier” because it helps ensure comprehensiveness and fairness. Companies interested in joining EEMBC or the Multicore Association can contact me directly for more information.

Markus Levy is founder and President of EEMBC. He is also President of the Multicore Association and Chairman of the Multicore Developer’s Conference. Markus began his career at Intel Corporation, where he served as both a Senior Applications Engineer and Customer Training Specialist for Intel’s microprocessor and flash memory products. He was also previously a senior analyst at In-Stat/MDR and an editor at EDN magazine, focusing in both roles on embedded processors. He is the coauthor of the book Designing with Flash Memory and holds several patents related to flash memory architecture and usage as a disk drive alternative. He is also a volunteer firefighter.


Follow: Twitter Facebook LinkedIn YouTube