01 March 2010 - 3.5 user guide eclipse intellij netbeans maven download nightly forum bugs blog sf.net eviware


Eviware Logo

soapUI Pro Web Service Coverage

soapUI Pro introduces the unique concept of Web Service Coverage, which dynamically allows you to analyze how much of a given SOAP or REST Web Service is being covered by your Functional Tests, MockServices, Mock Clients and SOAP Monitor clients and targets. This document aims at giving a brief overview of the concepts of Web Service Coverage, with links to the corresponding documentation for each Coverage point in soapUI. The best way to learn is to play around with it!

Start soapUI Testing
Did you know?

As the only test tool in the world, soapUI Pro shows you how much of your Web Services you have tested

This will ensure your test quality. Try it out and see how fun it is.

Try soapUI pro to soap test

Web Services Coverage

Traditionally in environments such as Java or .Net , coverage is used to measure how much of the actual code base that is being used during a given execution scenario (usually during Unit Tests). The returned value tries to give an estimate of how much of the code base that is actuality being used or tested.

Web Services in soapUI are defined by a WSDL contract or REST model (with or without a WADL), which specifies the operations, messages, faults, representations, etc... that make up the web services functionality. For each message there is generally a finite amount of defined elements and attributes that can be used to convey the message content, divided among message bodies, headers and faults.

The Web Service Coverage functionality analyzes each messages contract definition and then compares all corresponding messages during for example a TestRun to calculate how many of the defined elements or attributes that were actually used during the tests. The result is in soapUI presented as both a percentage and a visualization of the actual message; "Message Coverage"

Further, when running functional tests, soapUI analyzes the performed XPath assertions, and will try to calculate a corresponding value of how many of the defined elements or attributes that have actually been asserted. This result is added to the Message Coverage presentation and called "Assertion Coverage".

Coverage Visualization

In all situations where Coverage can be calculated in soapUI, a Coverage Panel is available with the following basic layout:

  • A toolbar at the top for enabling/clearing Coverage, setting Coverage Options and exporting a Coverage Report
  • A Coverage TreeTable to the left containing the same items as the containing project together with calculated coverage percentage/color bar for the item.
  • Two inspectors to the right containing either a Coverage Visualization for the selected operations request or response ("Message Coverage") or the actual Message Content for the selected request or response in the tree.
  • Optional Inspectors at the bottom, depending on measurement point:
    • Run Log - Shows a log of the
    • Assertion Results : available when running Test Coverage; contains a table of the performed XPath Assertions during the tests. Select a TestSuite, TestCase or TestStep to narrow down the assertions, or select an assertion in the table to navigate to that TestStep assertion in the tree.

An example of WSDL Coverage

The Coverage Tree

The Coverage Tree displayed to the left is divided into 2 parts:

  1. The Interface Part displays the current projects REST Services and SOAP Interfaces with their respective children. For each SOAP operation the defined Request and Response header/body/faults/attachments messages are shown. The displayed Coverage values are aggregated "upwards" in the tree, for example in the image above, the search Operations' Coverage is 8/9, which is the aggregate of the corresponding Request and Response message Coverage values. The values show that the Response Faults was not covered during the Test Run (0/1).

    For each REST method the defined Parameters and Representation body messages are shown.
  2. The TestSuite part displays the current projects' TestSuites, TestCases and relevant TestSteps. Under each TestStep that was the source of a Message Exchange during the Test Run, there will be Message nodes corresponding to the sent messages. Selecting a messages request or response node and then selecting the right Message Content inspector will show the actual content of the message (see image below). Selecting the Message Coverage inspector will instead show that messages' coverage value.

Well testedness of a SOAP Message

Double click an item in the tree will try to open that items corresponding editor/viewer

Assertion Coverage

As mentioned above, when running functional Tests, all performed XPath assertions are analyzed and a coverage value measuring how large part of the Web Service contract that was actually asserted. Assertion Coverage is displayed in dark-green in both the Coverage Tree and the Message Inspector. In the image below, the Message Coverage Inspector shows that the item and contained values have been asserted by an XPath expression, the assertion can also been seen in the Assertion Results Inspector at the bottom.

Assertion Coverage of a Web Service Response

Coverage Options

The Coverage Options dialog available from the toolbar allows you to set the following options:

  • Count Empty Values - select if empty elements and attributes should be regarded as covered
  • ? Values - select if elements and attributes containing '?' should be regarded as covered
  • Excluded Elements - a list of element names in the format "{" + Namespace URI + "}" + local part which should be excluded from coverage calculations.
  • Multi Values - a list of element names in the format "{" + Namespace URI + "}" + local part=comma-seperated list of values which will count coverage for the specified values. For example if you have a weekday element you could enter {http://my.namespace}weekday=mon,tue,wed,thu,fri,sat,sun and coverage will be calculated for the complete set of values.

Coverage Options in soapUI Pro

Coverage Measurement Points

soapUI allows you to measure coverage in 3 distinct scenarios:

  1. For Functional Tests - for measuring how much of your Web Service contract is actually covered by your tests. The corresponding Coverage Panel is available at the Project, TestSuite and TestCase level.
  2. For MockServices - for measuring both how much of your WSDL is being mocked and how much of your WSDL is being utilized/sent by the Mocks clients.
  3. For SOAP Monitors - for measuring how much of your WSDL is being used in a real life scenario.


Next: Standards and specifications supported by soapUI