Test SMI Implementation

Perform testing on each class

This step should be part of unit testing and final testing.  For unit testing, a provider developer can use a command line tool to execute the operations during development. The following items can be tested for each CIM Class that has been implemented (i.e. provider created).

  • Operations
    • Verify each operation supported by the provider works as designed
      • test each operation using all combinations of optional arguments
      • test each operation with invalid argument values to ensure proper error handling
    • Verify each operation not supported by the provider returns proper error code(s)/standard message
  • Properties
    • Verify that all Mandatory properties have non-null values
    • Verify conditional values, by testing condition and if met, validate value
    • Verify that each property has the proper value
  • Methods
    • verify method behavior
      • For Example, if creating a storage volume, verify the volume is created
    • verify input argument(s)
      • if a mandatory input argument is missing, method should return appropriate error
      • if an optional input argument is missing, method should succeed with proper defaults
      • if an input argument has invalid data, verify the proper error is returned
    • verify output argument(s)
      • verify output arguments are non-null when required
      • verify output arguments have the proper value
    • if the method supports a job, verify job control is properly handled
  • Associations
    • Verify that the proper instances get returned during associators/references
      • For example, starting from an instance of ACME_DiskDrive, the test verifies that instances of ACME_PhysicalPackage are returned using the CIM_Realizes subclass (e.g., ACME_DiskDrivePhysicalPackage) association.
      • Traversing the association from both directions must be tested
    • Verify Cardinality. The cardinality is either 1:1, 1:many or many:many. Verify that the cardinality is correct for the instances returned.

See Development Tools for the list of general purpose WBEM client tools that can be manually used for this purpose.

Test indication handling

The SMI Specification uses lifecycle and alert indications to notify a SMI Listener that an event has occurred. Lifecycle indications are typically used for configuration changes; e.g. when a storage volume is created or deleted. Alert indications are used for other conditions such as power supply failure. To be notified, an SMI Listener must subscribe for an indication by performing of the following steps:

  • create an instance of CIM_IndicationFilter
  • create an instance of CIM_ListenerDestination
  • create an association instance of CIM_IndicationSubscription

For example, in the Block Service Package, the SMI Listener must be notified when the size of a storage pool changes. The specific indication filter defined in the SMI Specification is "SELECT * FROM CIM_InstModification WHERE SourceInstance ISA CIM_StoragePool AND SourceInstance.CIM_StoragePool::TotalManagedSpace <> PreviousInstance.CIM_StoragePool::TotalManagedSpace". The SMI Client creates an instance of CIM_IndicationFilter containing this filter string. Next, the SMI Client creates an instance CIM_ListenerDestination that contains information about where the indication will be sent. The SMI Client then sends a request to the SMI Implementation to create an instance of CIM_IndicationSubscription, which references the newly created instances of CIM_IndicationFilter and CIM_ListenerDestination. Now the SMI Implementation knows that a SMI Client has subscribed to be notified when the event specified in the CIM_IndicationFilter instance occurs. When the size of a storage pool changes, the SMI Implementation will send an instance of CIM_InstModification, which contains an instance of CIM_StoragePool before the TotalManagedSpace property changed and an instance of CIM_StoragePool after the TotalManagedSpace property changed. The SMI Client can examine both instances to determine what action to take.

Using the same example, to properly test for this event, the indication test must subscribe using this indication filter. Then the test must change the size of a storage pool to force the event to occur. The indication test verifies that the proper notification is received.

For alert indications, a fault injection mechanism must be used to simulate or force the event. For example, to cause an alert indication to be sent for a fan failure, the fan must be forced to fail or at least appear to have failed.

See Development Tools for the list of general purpose WBEM client tools that can be manually used for this purpose.

Perform scalability, boundary condition, stress and performance testing

Two forms of scalability testing exist. The first type verifies that the SMI Implementation can handle a large number of objects. The other type verifies that the SMI Implementation can handle a large number of concurrent requests from one or more SMI Clients. Some profiles are used to manage a large number of resources. For example, the Block Services Package manages storage volumes. On some storage array products, many thousands of storage volumes can be created. For such SMI Implementations, testing verifies that a large number of storage volumes can be managed without causing product failures or unacceptable performance. Scalability tests are run against maximum configurations for this purpose. The other type of scalability testing involves forcing the the SMI Implementation to perform numerous operations at the same time. As an example, a test can issue 100 requests to create a storage volume at the same time.

For a particular resource, a product specification defines the maximum number allowed. For example, a array product might only allow 500 volumes to be created. Boundary condition testing would attempt to create 501 volumes to ensure that the last volume cannot be created.

Typically, a product specification defines the expected performance for certain operations and configurations. Performance tests verify that the SMI Specifications meets the product requirements. For example, if the product specification might require that 10 volumes to be created within 5 minutes or that the information for 10,000 storage volumes must be returned within 1 minute. A performance test verifies that these product requirements are met.

Some SMI Implementations use a cache to meet product performance requirements. When a cache is used, testing is required for each property that is cached. The test verifies that the cache contains accurate data; e.g., when 1000 storage volumes exists, the cache must have accurate information about all 1000 storage volumes. When the underlying configuration changes, then the cache must be updated properly. Otherwise, a SMI Client will receive inaccurate information. Depending on the cache architecture and product configuration, populating the cache data might take a long time. Cache testing verifies that the population times meet product requirements. For example, if the product requires that populating the cache for 100,000 volumes should only take 5 minutes, then the cache test verifies that this product requirement is met.

Perform profile level testing

Testing at the profile level is required to ensure proper interaction with a SMI Client. Profile level testing requires using the classes in a profile in the manner described by the SMI Specification, just as a SMI Client would do. For example, to test the Array profile, the profile tester would find the instance of CIM_ComputerSystem representing the array. From there, the profile tester would traverse the CIM_HostedStoragePool association to find all instances of CIM_StoragePool on the array. Each profile class would be tested similarly.

The profile test verifies that instances for all mandatory classes in the profile, as defined by the SMI Specification, are returned. For conditional profile classes, the profile test checks for the condition. If the condition is met, then the profile test verifies that instances of the conditional class are returned. The profile test will also attempt to retrieve instances of optional profile classes, but will not report an error if none are returned for a particular optional class.

For any returned instance, the profile test verifies that the instance contains the proper values as defined by the SMI Specification. The value of a property may depend on the value of another property. Depending on the profile and the property, testing for the allowed values of a property can be a complicated task.

See Test Tools for test suites that can be used for profile level testing of an SMI Implementation.