IP1 Evaluation: Did I get what I wanted?

What did I want?

From my statement of delivery I was hoping to achieve several goals and answer multiple questions about my concept with this prototype iteration. The big thing I was hoping to find out was how users would interact with the prototype lock and whether they would spin the dial or rotate it slowly point by point. Additionally I wanted to explore:

• Difficulty of the locks
• Feedback and if audio was sufficient by itself to support the interaction
• Difficulty Vs Reward and users feelings after opening the lock

How I go what I wanted?

I used an online survey to capture testers feedback immediately after their experience with the prototype. I also made a conscious effort to move away from testers while they undertook the survey, I learnt from the previous session (as both a tester and testee) that survey responses will be biased if they are observed. So I endeavored to make the survey experience as anonymous as possible given the classroom context.

I also observed each tester who participated and took notes in about the following aspects:

• Number of times unlocking physical lock
• Time taken to unlock physical lock
• Method used to unlock digital lock
• Preference and Approaches to unlocking
• Number of digital locks opened in 5 minute period
• Time taken per lock

I was also going to observe the number of times the reset button was pressed but due to technical issues with the prototype this number would have been heavily inflated and so was not recorded.

Results

Survey Findings

The results from the first two questions were very encouraging with all testers responding yes to both questions as shown below:

Question 1 Results

Question 2 Results

This was a relief as one of my reflections from my first testing session was that I did not stick well enough to the testing process which effected the quality of feedback I received. This time I actually brought a script which I ran through with each tester and provided a full overview of the session. While this made the sessions run a little long (only 6 participants) I feel that I helped the overall quality of results.

The results from question three helped to further understand the audience.

Question 3 Results

From this results as the vast majority of users had a reasonable level of familiarity with combination lock systems, I have decided to stick with this method of controller for the physical prototype iteration. I was initially concerned that the control would not be intuitive but this result shows that it should be feasible.

Question 5 and 6 were designed to look at the interaction method of the lock and how well it mapped to the physical lock. 

Question 5 Results

This result will be explained more in the observation section, but it became clear that the digital lock should have had an inverted control system (right arrow increase lock number, left arrow decrease lock number) rather than a direct mapping of the interaction.

Question 6 Results

Despite the dial not rotating as the majority of users expected there was a clear trend that users believed that the both locks controlled in a similar manner and opened in a similar way. This was the goal of the prototype and helped to solidify the design choice between a free spinning lock and a discrete feedback lock. 

Questions 4 and 7 were designed to evaluate the difficulty of opening the locks and how well the two (physical, digital) mapped.

Question 4 Results

Question 7 Results

I was glad to see that these two graphs overlapped to a large extend with both locks representing a similar level of difficulty. I was surprised to find that the physical lock where you are given the password was so difficult and the digital lock was perceived as easier to unlock. In hindsight a question about how appropriate the difficulty level was, should have been added to the survey.

Question 10 sort of answers this question by asking how long the passwords should be.

Question 10 Result

It is interesting that while testers generally rated the lock as difficult the no one thought that the passwords should be made shorter. While the majority think the 3 key is correct i was surprised that others opted to increase the number as well. This is something that can be further developed in additional prototypes with lock difficulty scaling and password numbers increasing as levels are completed.

Question 8 and 9 were looking to see how effective the prototype was at providing feedback to users on how the unlocking process was progressing.

Question 8 Result

Question 9 Result

This result is somewhat confusing as I was expecting question 9 results to be worse than question 8 results if testers found the feedback mechanisms were lacking. Perhaps this was due to my explanations providing too much detail in the instructions of how the system worked to testers. I still think a secondary visual feedback channel will need to be added for the next prototype to help users through the unlocking process and let them know which direction they should be turning the dial.

This blog post is already over the TLDR status, so I will leave the final evaluation and the observation results for a separate post.