The actual last post

Testing Session Observations and Qualitative Feedback

Looking at the users answers to survey question 5: Please finish this sentence, I accidentally turned the dial the wrong way when...

It is clear that users are running into this problem when the first begin opening a lock rather than when they have to change directions during the unlocking process. One user also remarked that most traditional locks will turn clockwise first where as the digital prototype turns anti-clockwise to begin. Again this could be another area for improvement in the future when experimenting with different control schemes.

Feedback from survey question again suffered from the ambiguity of 'visual feedback' without specifying either the turn indicator or the password progress system. Regardless users thoughts on the entirety of the right panel were largely encouraging. Indicating that the confirmation of password points was essential and am important supplement to the audio system. Users also pointed out that the arrow was not helpful initially as they were unaware it responded to the arrow key they needed to press. Suggestions for a call to action or tutorial system were again made and this will look to be implemented in future prototypes.

Observations

Based on my observations of the testers, on average most users opened two locks. One when they were completing their assigned task of testing the reset function and once afterwards when they had time to play before moving on. After the open of the initial lock using the reset method, most users were able to open the second lock in under 30 seconds.

It was also clear from observations that as users became more comfortable with the control system they moved from a simple tap system of pushing the key in once to the holding method to make the lock dial rotate more quickly. One user even discovered that she could open a lock simply by spinning in one direction to the correct number, resetting, hitting that point and then changing direction. Using this method they were able to open locks very easily and quickly, however, this was unintentional and may need to changed slightly to ensure users are skill relying on precision rather than brute force to open locks.

Finally most users interacted with the reset button six times or more as they ran through the initial task and the opening of additional locks. It seems clear from this testing session that the stability changes made to the code were successful and both the reset and new lock buttons are functioning correctly.

Its the Final Countdown

Quantitative Testing Session Results

Another testing session bites the dust and its time to look at the results of the latest iteration. In this post I will look at the survey results and how the relate to the goals for this prototype and breakdown the responses from my five victims. This will be looked at the quantitative data recorded with a separate entry for session observations and quantitative results.

Of the testers 4/5 had prior experience with the Street Survivor prototypes which was important for testing the effectiveness of the visual feedback relative to the physical method used in prototype two. Of the four that had previous experience 75% said that they found that the locks were easier to open with this prototype.

Interesting though a significant number of users still had trouble rotating the dial in the correct direction, despite the visual feedback.

Results for Question 3.

This trouble was also reflected in question four with most two users reporting they had repeated troubles. Rotating the dial in the wrong direction more than five times.

Results for Question 4

This is an interesting development as again it seems that users are focusing their attention entirely on rotating the lock rather than the supplementary information provided. Even a result of 3-4 times could be considered a failure for smooth operation of the controls. From this result further refinements would be to invert the control scheme or alternatively rebuilt a high quality physical system.

Despite their troubles users still reported that the visual aids were helpful to their playing of the game.

Results for Question 8

Again this seems counter intuitive, but in hindsight an important design flaw may be behind this result. Both the password progress system and the visual indicator were classified as 'visual feedback' systems in both the survey and testing session wording. These will need to be separated in future testing sessions and potentially even having the code modified to include only one of the features.

Results for Question 7

The results from question seven make it clear that the password progress box was helpful in improving users experience and may have been the primary reason for users finding the locks less difficult to unlock. Overall the difficulty seems to have stabilized to an adequate level where it is sufficiently challenging as the results from question six suggest.

Results for Question 6

From these results it seems clear that the added features have not lead to an improvement in users ability to rotate the dial in the correct direction. Further work will need to be done in this space to ensure that future builds are more usable. It also shows that the locks are easier to open and potentially reduces the reward felt by users when opening a lock.

Interactive Prototype 3 Facilitator Script

Welcome to this testing session and thank you for participating and providing us with your feedback today. I will be here to observe the testing and will be taking notes but feel free to ask me at any time for clarifications if you have any questions.

I just want to run you quickly through the testing process today firstly. We are testing the Visual Feedback elements of the Street Survivor game.

• So the Direction indicator and the Password progress display

For this testing session there will be three phases. We are currently in the first phase, now then there will be some tasks with the digital prototype and finally a survey at the end.

Before we start phase two so that you can familiarize yourself with the way a combination lock is supported to work and how to open one I have brought one along a for you.

When you are ready we will move into Phase 2 and get started with the digital prototype. Where we have a few tasks for you to complete.

1.  Spin the digital lock dial to the first password point, once the visual feedback presents. Reset the lock.
2.  Spin the digital lock dial to the first password point through to the second point. Again once feedback presents. Reset the lock.
3. Attempt to open the lock by moving through all three password points.
4. Generate a new lock and follow steps 1-3.
5. Users are given 3 minutes to open as many locks as possible.

- Following this will be a short survey for you to fill out privately and the observation of the session will cease.

INSTRUCTIONS FOR PHYSICAL LOCK

- Click this around a few times until you are familiar with the feel of it.

INSTURCTIONS FOR DIGITAL LOCK

- If at any point you make a full rotation of the lock without hearing any audio feedback or seeing any visual feedback, please reset the system and advise myself as a bug has occurred.

- The reset button will reset your progress on the current lock password. While the new lock button will generate a new lock with a new password.

Observations:

      Number of digital locks opened.
Time spent per digital lock.
Number of times controller turned in the wrong direction, when starting a lock.
User control patterns (slow click or fast spin)
Number of times reset lock button is pressed.

Survey Time Sadness

Survey Results

Despite what the title suggest this is not another doom and gloom post ^.^

Responses from the survey data were largely positive. With encouraging feedback about the concept and the viability of the physical lock mechanic. With one user stating that "I still like the idea, and the game mechanics is fine". This adds to the continued belief that the core concept is strong and the resulting game mash-up is both interesting and fun. Test suitability of a physical lock style controller was one goals of the session.

Responses to survey question three also revealed important information about the difficulty of understanding the controller interaction. With the majority of testers having a clear understanding of how the lock should function in the game context.

Q3 Result

However, the results from question four were largely unexpected given the technical issues faced during the prototyping session.

0 Representing unsure 5 Representing Completely Sure.

I feel like this result was heavily influenced by way users focused their attention on the screen rather than the physical controller. This was observed during the testing sessions and the survey responses to question 5 corroborate the evidence.

From these findings it is clear that the physical lock does not need to accurately represent the lock on the screen. There is no need for the physical lock to have the same number of points it just needs to ensure it has the same feedback. 

Q5 Results

Also encouraging was the result from question 6 which showed that the prototype provided acceptable levels of tactile feedback. Testing the feedback of the controller was one of the major goals of the testing session.

Where 1 is completely dissimilar and 5 is completely similar

However the biggest finding coming out of the session was that the actual controller was physically difficult to rotate. My obsession with making something that was sturdy backfired as the lock would only rotate with a significant amount of force. This may have been due to the fact people thought they would break the prototype but it is something that will be looked into for the next iteration.

Other tester frustrations and pain points included:

• Changing directions
• Lack of sensitivity
• Points missed
• Points skipped 
• Level of audio feedback (Too quiet)

TLDR Testing Session Goals and Outcomes Recap:

Testing Goal: Is a physical combination lock style controller suitable for the game?
Outcome: Yes, though the prototype needs to be refined

Testing Goal: Does the prototype provide an intuitive control system?
Outcome: Yes, the control scheme is easily understandable

Testing Goal: Does the physical controller dial accurately represent the digital lock dial?
Testing Goal:  By how many points to the two dials differ?
Outcome: Unimportant, users spend majority of time looking at screen to show lock position.

Testing Goal: Does the physical lock provide enough tactile feedback when moving from point to point?
Testing Goal: Does the physical lock accurately represent a real lock?
Outcome: Yes and yes. If anything it provides too much feedback and requires too much force to use.

Testing Goal:  Does the physical lock make the game more or less difficult?
Outcome: Unclear, at this stage the current prototype is more difficult due to the force required to rotate. But the actual control system could be harder or easier than keyboard.

Testing Goal: Can the prototype complete a testing session without repair?
Outcome: Definitely not in its current state.

Testing Goal:  How many locks can the prototype open without issue?
Outcome: Zero

Lock Life - Testing Session

The day of the testing session started with great promise. A bright sunny day, a pleasant bus ride. Simple set up. No strange looks from others.

Babys first bus ride

Then it was time to get serious and get into the testing. From the outset it became obvious that there was not going to be a lock successfully opened. Running a testing session and observing the users soon turned into holding parts of the prototype together, explaining bugs, and making running repairs in an attempt to reach the mythical five tester mark. Which I managed to achieve with video evidence to prove it!

User Test 1.

User Test 2.

User Test 3.

User Test 4.

User Test 5.

Despite the issues with the prototype, I received and observed positive feedback on the prototype when it WAS working. When the wheel was turning correctly there was definite feedback, the on screen lock was turning as intended. The major sticking point throughout the testing session was changing of directions, when the lock needed to move from anti-clockwise rotation to clockwise rotation.

Upon inspection it was found that the pointer was continually getting stuck on nails and just sticking into them rather than sliding past them when a change of direction was made. This problem meant that my observations were largely pointless as a grand total of 0 locks were opened with all participants taking the full five minutes (or until they got frustrated) to attempt a single lock.

Additionally the dial face on the controller came loose in one of the user session which meant that it was not accurately representing the points underneath it. And the lock got stuck on a point so many times I lost count.

Hopefully its not so doom and gloom when we hit the survey results.

PS: Shout out to all my testers ^.^ thank you so much for your feedback and perseverance.

Time for HardHats

After my initials constructions I was sufficiently satisfied that my current design and set up would work at full scale. So it was time to get out the saw, and cut some MDF.

Thanks Saw Horses

Having some Density issues, with the large nails

Cant go back now.

Good Thing it fits

The base to mount the wheel onto

After spending the best part of the morning with a saw in one hand and a hammer in the other it was time for the moment of truth. Fastening the lazy Susan hardware, I only bought one of these so it was the point of no return from here on.

We have life!

IT FITS!

Time to test

With the wheel on and correctly rotating, I thought the hard part was behind me. Little did I realize that I needed a way to fix the pointer to the base, which would still allow it to rotate so it would pass through each point. BUT, not have it rotate too far that it might miss some of the points or fail changing direction.

The Twin Foam Duct-tape house

Nails with foam reinforcement

The initial set up with foam would work for a small period of time, providing enough force to keep the pointer hard up against the wheel to make sure it would contact all the points. This problem was exemplified by the fact some points were alot further in due to nails on all sorts of angles. The final solution was to have two nails either side of the pointer (As they were stronger and would hold it in place longer) to ensure contact with every point.

Finally it was time to box it up and make it look pretty (thorough testing would have to wait. Many fingers were crossed)

Cozy Foam Home

Always good to have a spare Beer Carton

Before the Facelift

Paint Shed

Final Test

So there it is, a little black box of hopes and dreams. Ready for its time to shine at the testing session.

MidSemester Break(ing) My Prototype

MidSemester break provided a great opportunity to actually go outside and hunt around for materials that would be more suitable to create my prototype out of. After multiple trips to Bunnings and an emergency visit to Big W for a compass set (like high-school all over again) I was ready to get into construction.

Have to make sure you get enough nails. The lazy Susan assembly was the essential part

MDF and Foamcore. Test with one build with the other.

The break also gave me some time to research how I would build it, rather than just blindly rushing into it. I had a quick (in hindsight probably too quick) look at WikiHow and U-Createcrafts tutorials for creating prize spinners. Which heavily inspired my final design.

Then It was time to get building, then testing. Then building a little bit more, then testing a lot more. Then back to building... you get the idea.

Make sure your earthed!

A Quick Test. Would the nails wired together both be earthed?

Testing the MakeyMakey inputs for left and right.

With the first tests completed, I had established that nails would work, I would just need to wire them together, but once this was done the whole wheel could be used as the earth. Meaning that the user would not have to hold onto anything else to make the prototype work. Now it was time to include some more points on the circle.

Initial fail attempt. Which resulted in Compass purchase.

Woah Were Half Way There!

Better check it again to make sure it works at a larger scale.

This is where one of the first lessons came up, and one of the major problems which continued to plague the design process. Needing to ensure that the points on the wheel were not only evenly spaced but the correct with from the center. Working by hand this proved to be difficult as nails dont always do what you want them to when a hammer is viciously applied to them.

I then had a quick experiment with different types of pointers which would make contact with the wheel. Whether to put them internally or externally. How they would be controlled?

Large Central dial

CHAOS

Quick test of having multiple pointers

In the end because of the difficulties in ensuring a consistent distance of nails, I decided it would be easier to have the pointer on the outside of the wheel and have it work more like a wheel of fortune style wheel.

End of Construction Day 1.