VMware VMmark

Open Standards & Platform Neutrality

VMmark is not a commercial or proprietary product, but rather a product of our commitment to the development of open standards for virtualization benchmarks. As such, the VMmark software is agnostic towards individual hardware platforms and virtualization software systems so that users can get an objective measurement of virtualization performance.

VMware is working with the Standard Performance Evaluation Corporation (SPEC®) - a non profit organization that establishes, maintains and endorses standardized benchmarks - and members of the SPEC Virtualization subcommittee to develop standard methods of comparing virtualization performance for data center servers.

Other current participants include AMD, Dell, Fujitsu Siemens, Hewlett-Packard, Intel, IBM, Microsoft, Red Hat, Sun Microsystems and SWsoft. For additional information, visit the SPEC Virtualization Subcommittee Web site.

Application-centric Benchmarking of Real-world Workloads

VMmark uses workloads representative of those applications most often found in the data center, such as email servers, databases, etc. VMware has worked closely with its partners to design and implement the benchmark across various software and hardware platforms, and has also gathered extensive customer feedback to understand how these applications are typically used in virtualized environments.

To measure performance, VMmark leverages well-understood, existing benchmarks that customers are already familiar with.

VMmark incorporates 64bit versions of three of the VMmark workloads, Java Server, Database Server, and Web Server to reflect the growing use of 64bit applications and OSs.

Unique Tile-based Implementation

The unit of work for a benchmark of virtualized consolidation environments can be naturally defined as a collection of virtual machines executing a set of diverse workloads. The VMmark benchmark refers to this unit of work as a tile.

The total number of tiles that a system can accommodate provides a coarse-grain measure of that system's consolidation capacity. This concept is similar to some server benchmarks, such as TPC-C, that scale the workload in a step-wise fashion to increase the system load.

High-precision Scoring Methodology

VMmark allows for the integration of the different component metrics into an overall score. Once a VMmark test completes, each individual workload reports its relevant performance metric. These metrics are collected at frequent intervals during the course of a run. A VMmark benchmark test is designed to run for at least three hours with workload metrics reported every 60 seconds.

After a benchmark run, the workload metrics for each tile are computed and aggregated into a score for that tile. This aggregation is performed by first normalizing the different performance metrics such as MB/second and database commits/second with respect to a reference system. Then, a geometric mean of the normalized scores is computed as the final score for the tile. The resulting per-tile scores are then summed to create the final metric.

This approach helps users measure the virtualization overhead of the individual application workloads, as well as the scalability of the entire system.