When using the method to test a screen-based project, the ‘wizard’ and participant are often kept in separate locations so as not to break the ‘illusion’ of a fully functional prototype. During testing, the wizard will simulate interactions in response to participant actions, which are usually relayed to the wizard via screen sharing or other video software. The wizard may assume a variety of roles, such as simulating an ‘intelligent’ system, course-correcting and potentially overriding user decisions as well as simulating sensory data to ensure the user experience feels complete. While separation of wizard and participant makes sense for screen-based projects, when using the WOz method to test tangible or physical media projects it is often necessary for the wizard to be physically present to ‘pull the strings’ of their project. In these situations, ‘sleight of hand’ can be used when demonstrating the interactions of a tangible media project, for example, manually triggering an LED to blink when a user performs a particular interaction, implying that their action has resulted in the LED blinking.

Whether you use the WOz method for screen-based or tangible media interaction projects, consistent responses from the wizard are fundamental to ensuring the ‘illusion’ you’ve created isn’t broken by your participant. Ensure timing, patterns of response and the underlying system logic are all consistent and consider rehearsing a few use-case scenarios before putting on your wizard hat!

Activity

Before you start

With your project team, agree on your aims and goals and develop a list of questions you want to test out with your participants. Organise these questions in order of importance and use this list to inform how you produce your prototype. You may be testing a specific part of the project or an early version of the entire project – either way it’s important to be realistic about what you can achieve in the time you have available. Produce a prototype or series of prototypes that make the most of readily available materials, such as cardboard, paper and other lo-fidelity prototyping formats. Consider how you can augment these prototypes using smartphones and laptops. Identify and recruit your participants – ideally these would be prospective users. If you don’t have access to these people, friends and peers are a good substitute as this gives you an opportunity to gain informal feedback on your project while also testing out how to run a WOz testing session. Prepare your prototypes, assign roles and rehearse your testing scenarios, making changes to actions, responses and roles until your intended experience has been created. Finally, prepare your testing space and ensure that it meets your team’s requirements.

Activity steps

1. Begin your scenario by introducing participants to the project and ensure they’re sufficiently briefed on the project with enough context to understand what’s expected of them. Enact the scenario – the wizard/s will simulate functionality in the prototype to varying degrees and remain hidden from the user.
2. While the wizard/s operate the prototype, observers will record a list of findings such as errors, ideas, issues and anything relevant to the aims of the testing session.
3. If possible, make some adjustments in real time, where the observers share brief insights with the wizards, make changes to the scenario as relevant and then integrate these changes into a new scenario and begin the test again.
4. Once all of your scenarios have been enacted, reveal the wizard/s to the users and have a brief discussion with them about what did and didn’t work. Record any of these findings and add these to the points taken by the observers.
5. In your team discuss the findings from observers, users and anything else uncovered throughout the testing process. Reflect on and integrate relevant findings into future iterations of the project.

References

1. Kelley, J. 1984. ‘An Iterative Design Methodology for User-Friendly Natural Language Office Information Applications’. ACM Transactions on Information Systems (TOIS) 2 (1): 26–41. https://doi.org/10.1145/357417.357420.

• The evaluation environment and participants/users are selected – these should both mimic their intended ‘real world’ context as closely as possible
• Depending on the project, participants are fitted with head or body mounted video recording devices; or devices are placed in/around the evaluation environment in such a way that they do not intrude or distract from the participant experience.
• Participants then experience the project within the evaluation environment, which is recorded using the equipment.
• After the participant experience, recording equipment is removed from the participant and video footage is transferred to a playback device (such as a laptop).
• The participant is then interviewed while the video footage of their experience is played back to them.
• This interview is also video recorded, ensuring that the participant and screen are both in frame as you’ll want to refer to what on-screen footage the participant is discussing.

Activity

Before you start

In our experience, video-cued recall is well suited to gathering participant data with projects that are somewhat technically resolved, such as a working prototype. We’ve founded that using video-cued recall with functional prototypes takes some of the pressure off worrying about technical issues that might arise when working with early or lo-fidelity prototypes and allows the design or research team to focus on managing logistics. Some key questions to address before implementing video-cued recall with your project include:

• How many cameras will you need to effectively record users experiencing your project?
• Where and how will these cameras be placed?
• Are there ambient light and sound issues to take into consideration?
• How will you efficiently transfer these video recordings to a playback device?
• When interviewing your participants, how will these interviews be recorded?

We suggest running a mock test of your intended data collection setup to help work through any potential logistical issues and make adjustments as required.

Activity steps

1. Setup your evaluation environment, prepare your interview questions and recruit your participants.
2. If relevant, brief your participants on what is expected of them and provide enough context to ensure that their experience of your project accurately reflects how they would experience it in a ‘real-world’ context of use.
3. Begin your evaluation. Depending on your project, this could take anywhere from 5-30 minutes. While participants are experiencing your project, you may take this opportunity to make your own observations and note these down for discussion in the following interview.
4. Transfer video of the participant/s to the playback device.
5. Conduct the interview and use your questions as a loose structure. These questions should address any key concerns, guided by your project aims, while also allowing the participant an opportunity to expand upon the discussion. Use the video of the participant experiencing the project to address specific usability and experiential elements of the project. Ask them what they were thinking and feeling at specific, relevant moments of their experience – this is where the video ‘cues’ their recall.
6. Ensure you keep copies of all video files – we suggest developing a rigorous archiving system as this will help you keep track of all your data.
7. Transcripts of these interviews, while valuable data, can be difficult to interpret without sufficient context. This is where the video recordings of the interviews become invaluable. Depending on how you intend to use your data, you might include additional context cues within transcripts or timecodes within transcripts to quickly navigate to specific sections of the interview video.

Notes

Students within the Creative Industries Faculty have access to the faculty Equipment Loan Centre (ELC), which is located on the ground floor of Z6 building at the Kelvin Grove campus. The ELC allow students to borrow out equipment to use for their studies, which includes DSLR cameras, video cameras, GoPro’s, tripods and more. If you intend to use video-cued recall to aid in your assessment work, discuss with your lecturer if borrowing from the ELC will suit your needs as this is a great resource available to you. Do keep in mind that equipment availability is subject to demand and loan periods vary depending on student and unit needs.

Creative industries equipment loan centre

• Location: Z6 Block, Level 1
• Email: ci-elc@qut.edu.au
• Phone: 07 3138 8121

References

1. Miller, Anne. 2004. ‘Video-Cued Recall: Its Use in a Work Domain Analysis’. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 48 (15): 1643–47. https://doi.org/10.1177/154193120404801503.