The actual last post

Testing Session Observations and Qualitative Feedback

Looking at the users answers to survey question 5: Please finish this sentence, I accidentally turned the dial the wrong way when...

It is clear that users are running into this problem when the first begin opening a lock rather than when they have to change directions during the unlocking process. One user also remarked that most traditional locks will turn clockwise first where as the digital prototype turns anti-clockwise to begin. Again this could be another area for improvement in the future when experimenting with different control schemes.

Feedback from survey question again suffered from the ambiguity of 'visual feedback' without specifying either the turn indicator or the password progress system. Regardless users thoughts on the entirety of the right panel were largely encouraging. Indicating that the confirmation of password points was essential and am important supplement to the audio system. Users also pointed out that the arrow was not helpful initially as they were unaware it responded to the arrow key they needed to press. Suggestions for a call to action or tutorial system were again made and this will look to be implemented in future prototypes.

Observations

Based on my observations of the testers, on average most users opened two locks. One when they were completing their assigned task of testing the reset function and once afterwards when they had time to play before moving on. After the open of the initial lock using the reset method, most users were able to open the second lock in under 30 seconds.

It was also clear from observations that as users became more comfortable with the control system they moved from a simple tap system of pushing the key in once to the holding method to make the lock dial rotate more quickly. One user even discovered that she could open a lock simply by spinning in one direction to the correct number, resetting, hitting that point and then changing direction. Using this method they were able to open locks very easily and quickly, however, this was unintentional and may need to changed slightly to ensure users are skill relying on precision rather than brute force to open locks.

Finally most users interacted with the reset button six times or more as they ran through the initial task and the opening of additional locks. It seems clear from this testing session that the stability changes made to the code were successful and both the reset and new lock buttons are functioning correctly.

Its the Final Countdown

Quantitative Testing Session Results

Another testing session bites the dust and its time to look at the results of the latest iteration. In this post I will look at the survey results and how the relate to the goals for this prototype and breakdown the responses from my five victims. This will be looked at the quantitative data recorded with a separate entry for session observations and quantitative results.

Of the testers 4/5 had prior experience with the Street Survivor prototypes which was important for testing the effectiveness of the visual feedback relative to the physical method used in prototype two. Of the four that had previous experience 75% said that they found that the locks were easier to open with this prototype.

Interesting though a significant number of users still had trouble rotating the dial in the correct direction, despite the visual feedback.

Results for Question 3.

This trouble was also reflected in question four with most two users reporting they had repeated troubles. Rotating the dial in the wrong direction more than five times.

Results for Question 4

This is an interesting development as again it seems that users are focusing their attention entirely on rotating the lock rather than the supplementary information provided. Even a result of 3-4 times could be considered a failure for smooth operation of the controls. From this result further refinements would be to invert the control scheme or alternatively rebuilt a high quality physical system.

Despite their troubles users still reported that the visual aids were helpful to their playing of the game.

Results for Question 8

Again this seems counter intuitive, but in hindsight an important design flaw may be behind this result. Both the password progress system and the visual indicator were classified as 'visual feedback' systems in both the survey and testing session wording. These will need to be separated in future testing sessions and potentially even having the code modified to include only one of the features.

Results for Question 7

The results from question seven make it clear that the password progress box was helpful in improving users experience and may have been the primary reason for users finding the locks less difficult to unlock. Overall the difficulty seems to have stabilized to an adequate level where it is sufficiently challenging as the results from question six suggest.

Results for Question 6

From these results it seems clear that the added features have not lead to an improvement in users ability to rotate the dial in the correct direction. Further work will need to be done in this space to ensure that future builds are more usable. It also shows that the locks are easier to open and potentially reduces the reward felt by users when opening a lock.

Interactive Prototype 3 Facilitator Script

Welcome to this testing session and thank you for participating and providing us with your feedback today. I will be here to observe the testing and will be taking notes but feel free to ask me at any time for clarifications if you have any questions.

I just want to run you quickly through the testing process today firstly. We are testing the Visual Feedback elements of the Street Survivor game.

• So the Direction indicator and the Password progress display

For this testing session there will be three phases. We are currently in the first phase, now then there will be some tasks with the digital prototype and finally a survey at the end.

Before we start phase two so that you can familiarize yourself with the way a combination lock is supported to work and how to open one I have brought one along a for you.

When you are ready we will move into Phase 2 and get started with the digital prototype. Where we have a few tasks for you to complete.

1.  Spin the digital lock dial to the first password point, once the visual feedback presents. Reset the lock.
2.  Spin the digital lock dial to the first password point through to the second point. Again once feedback presents. Reset the lock.
3. Attempt to open the lock by moving through all three password points.
4. Generate a new lock and follow steps 1-3.
5. Users are given 3 minutes to open as many locks as possible.

- Following this will be a short survey for you to fill out privately and the observation of the session will cease.

INSTRUCTIONS FOR PHYSICAL LOCK

- Click this around a few times until you are familiar with the feel of it.

INSTURCTIONS FOR DIGITAL LOCK

- If at any point you make a full rotation of the lock without hearing any audio feedback or seeing any visual feedback, please reset the system and advise myself as a bug has occurred.

- The reset button will reset your progress on the current lock password. While the new lock button will generate a new lock with a new password.

Observations:

      Number of digital locks opened.
Time spent per digital lock.
Number of times controller turned in the wrong direction, when starting a lock.
User control patterns (slow click or fast spin)
Number of times reset lock button is pressed.