OUTCOMES
Going into the testing session I set up my laptop with my video ready to play and google form open ready to receive feedback. I firstly explained to users that I was looking for feedback on both the concept and the controller system before prompting them to view the video. I advised testers that they could pause rewind and scrub through the video as they needed and ask questions for clarification.Of the 6 users that observed my game there was only one that watched the video multiple times and asked questions for clarification. The majority of testers going straight through the video on the first run with no pause or rewind needed.
Aside from my personal observations there was also a survey which testers were asked to fill out after watching the video. The results from this survey are presented below. Please note when the bar is not visible it is because a score of 1 was received for the question.
The results are mostly positive with almost all users agreeing that they could easily understand the grounding of the concept. Likewise the majority understood the premise and purpose of the game, however there were two who agreed that the purpose was difficult to understand.
The results for question three and four are some of the most encouraging, with the majority of users strongly agreeing that the controls were not only understood from the video explanation but that the control method was actually interesting.
Questions five and six related to the actual quality of the prototype and the video production. Again the majority of scores are high (or low where question statement is reversed) suggesting that the prototype was of acceptable quality and was able to effectively communicate without serving as a distraction.
The final questions related to the testing process itself and the transparency of the method described in the statement of delivery. The results suggest that this is the area where most improvement could be made. Improving explanation to testers what exactly they should be testing will be imperative to retrieving relevant and useful feedback.
While there was a wide range of questions on the survey the purpose of the prototype was to communicate the concept and so feedback was sought specifically around this area. As well as on the appropriateness of the control system.
REFLECTION
I feel that the video format made for an effective media to communicate my concept to others. Again in hindsight I think that my video was about 30 seconds too long at 2 minutes. The rest of the video I am happy with, the voice over added to the theme of the video without being distracting. There was no feedback from testers that indicated it negatively affected their understanding of the concept. Also from the feedback it seems that the paper animations were appropriate and overall well received.
Overall I think the video prototype was a very useful step to take in the design process. Video was a good way to communicate across both visual and audio channels. The experiences of making this video has been helpful as I can see the importance of using simple audio and visuals to get the message across. You cannot just prattle on and on or the user will lose interest.
Finally I think that adhering to my testing protocol was the area that I need to most improve in. Making sure that I faithfully follow the process each time so that each user has the same experience will increase the overall effectiveness of my testing protocols. Hopefully this will result in more constructive feedback in future prototypes.
EFFECTIVENESS
For the purpose it was designed (to communicate my concept) the video was successful. The results showed that most users left with a good understanding of both the concept and the control system. However a recurring theme from the written feedback was that the elements of Operation were not clearly defined. This is due to the concept drawing more heavily on Street Fighter and not explicitly taking any gameplay from Operation (Only the tactile input system and idea of fine motor control). While I dont see this as a failure of my initial prototype it may be something to incorporate additional gameplay aspects in later prototypes of the game as it becomes more sophisticated.
Most of the feedback was around possible expansions to the game and deeper explanation of the current systems. After viewing other peoples videos, it can see that my method of describing the gameplay relied too heavily on audio, without enough visual support. However, as part of the interface prototype I will look to expand and show how the gameplay will look and feel.
CONSTRAINTS
Aside from my own skills in creating video content, there were a few constraints which impacted upon the video results. The limitation on how much video had to be actually filmed lead me to use the paper animation style. Additionally constraints in the testing environment meant that I was not able to gather as much feedback as I would have liked.IMPLICATIONS
From the feedback received am I confident that my core concept is solid and can be used to create further prototypes. However, I will be conscious of the need to explain gameplay elements clearly and explain systems more clearly in further prototypes.For future testing sessions I will ensure that I print a printed script for myself to follow to ensure that I am sticking to the testing protocol. Also I will consider shortening the length of the survey and changing the phrasing of the questions to ensure the testers are not getting confused or bored with the feedback system.
