Concept testing
Different testing methods will have different objectives, approaches and types of modelling. Four general types of testing are described in more detail:
- Exploratory tests
- Assessment tests
- Validation tests
- Comparison tests
ISO 9000 tests are also briefly summarised.
Exploratory tests
Carried out early in the development process during the fuzzy front end, when the problem is still being defined and potential solutions are being considered, preferably once the development team has a good understanding of the user profile and customer needs. The objective of the exploratory test is to examine and explore the potential of preliminary design concepts and answer some basic questions, including:
- What do the users think about using the concept?
- Does the basic functionality have value to the user?
- Is the user interface appropriate and operable?
- How does the user feel about the concept?
- Are our assumptions about customer requirements correct?
- Have we misunderstood any requirements?
This type of early analysis of concepts is potentially the most critical of all types of prototyping and evaluation, for if the development is based on faulty assumptions or misunderstanding about the needs of the users, then problems are almost inevitable later on. Data collection will tend to be qualitative based on observation, interview and discussion with the target audience. Ideally, the customer should be asked to use the product without training or prompting, to assess the intuitiveness of controls and instructions. Some quantitative measures may be appropriate, such as time to perform tasks, number of failures or errors.
Assessment tests
While the exploratory test aims to explore the appropriateness of a number of potentially competing solutions, the assessment test digs into more detail with a preferred solution at a slightly later stage of development. The main aim of an assessment test is to ensure that assumptions remain relevant and that more detailed and specific design choices are appropriate. The assessment test will tend to focus on the usability or or level of functionality offered and in some cases, may be appropriate for evaluating early levels of performance. Assuming that the right concept has been chosen, then the assessment test aims to ensure that it has been implemented effectively and answer more detailed questions, such as:
- Is the concept usable?
- Does the concept satisfy all user needs?
- How does the user use the product and could it be more effective?
- How will it be assembled and tested and could this be achieved in a better way?
- Can the user complete all tasks as intended?
Assessment testing typically requires more complex or detailed models than the exploratory test. A combination of analytical models, simulations and working mock ups (not necessarily with final appearance or full tooling) will be used.
The evaluation process is likely to be relatively informal, including both internal and external stakeholders. Data will typically be qualitative and based on observation, discussion and structured interview. The study should aim to understand why users responds in the way that they do to the concept.
Validation tests
The validation test is normally conducted late in the development process to ensure that all of the product design goals have been met. This may include usability, performance, reliability, maintainability, assembly methods and robustness. Validation tests normally aim to evaluate actual functionality and performance, as is expected in the production version and so activities should be performed in full and not simply walked through.
It is probable that the validation test is the first opportunity to evaluate all of the component elements of the product together, although elements may have been tested individually already. Thus, the product should be as near to representing the final item as possible, including packaging, documentation and production processes. Also included within validation tests will be any formal evaluation required for certification, safety or legislative purposes. Compared to an assessment test, there is a much greater emphasis on experimental rigour and consistency. It may be preferable for evaluation to be carried out independently from the design team, but with team input on developing standards and measurement criteria.
Data from a validation test is likely to be quantitative, based on measurement of performance. Normally, this is carried out against some benchmark of expected performance. Usability issues may be scored in terms of speed, accuracy or rate of use, but should always be quantified. Issues such as desirability may be measured in terms of preference or user ranking. Data should also be formally recorded, with any failures to comply with expected performance logged and appropriate corrective action determined.
Comparison tests
A comparison test may be performed at any stage of the design process, to compare a concept, product or product element against some alternative. This alternative could be an existing solution, a competitive offering or an alternative design solution. Comparison testing could include the capturing of both performance and preference data for each solution. The comparison test is used to establish a preference, determine superiority or understand the advantages and disadvantages of different designs.
ISO 9000 tests
ISO 9000 defines a number of test activities:
Design review
A design review is a set of activities whose purpose is to evaluate how well the results of a design will meet all quality requirements. During the course of this review, problems must be identified and necessary actions proposed.
Design verification
Design verification is a process whose purpose is to examine design and development outputs and to use objective evidence to confirm that outputs meet design and development input requirements.
Design validation
Design validation is a process whose purpose is to examine resulting products and to use objective evidence to confirm that these products meet user needs.
More ISO 9000 definitions are to be found here.
For more information, please contact:
T: +44 1223 764830
