Crystalim Testing

Usability Test Plan

Paula HairLauren Jenkins

Matthew GardnerAlex Talbott

Table of ContentsTable of Contents

Executive SummaryTesting Methodology

ParticipantsTrainingProcedure

Testing GoalsRoles

TesterParticipant

Usability GoalsCompletion RateError Free RateTime On TaskDeath Counter

QuestionsImpression QuestionsExploratory Questions

Initial Testing ExperienceFlowchartReferences

Executive Summary

This document will provide a review of the usability aspects of Crystalim. The process for this usability test will involve recruiting people that qualify as casual gamers, or people with limited knowledge of video games. The tester will be asked various questions regarding their experience playing the game as well as any impressions they received upon seeing the game for the first time.

Specifically, the use of Crystalim’s spirit world mechanic will be evaluated and analyzed. The evaluation will look at player reaction when utilizing the mechanic as well as role it plays in the game such as how frequent the mechanic is used.

Testing Methodology

Participants

There are 4 expected participants, 1 for each tester. The participants are casual gamers who aren’t very experienced. They will play through the game in its entirety while giving feedback for the usability. The participants do not have to have a set background and will be chosen based on availability due to the time crunch.

Training

No prior training is provided, as we want the tester to mold their impressions during initial gameplay rather than what they are told.

Procedure

Participants will use 15 inch MacBook Pros running in Boot Camp on the Windows side. The machines run Windows 7. After the participant boots the game, the tester will ask the impression questions while they are figuring out the Menu. Testers will sit in the room with participants and note their feedback. The testers will also time participants, logging the data throughout the test. The tester will not lead the participant in any way to ensure that the participant can illustrate whether or not the game is playable with no prior instruction. They will merely listen, ask for clarification when necessary and be sure that they understand the participant’s response. After the levels are completed, the tester will then ask the exploratory questions to really dig into the tester’s view of the game.

Testing Goals

The goals of the usability test are:● Determine any design inconsistencies between the game and the game design while

making note of the build notes document. Errors or inconsistencies may include:○ Navigation errors - failure to locate functions, excessive keystrokes to complete a

function○ Presentation errors - failure to locate and properly act upon desired information in

screens, displays of incorrect information○ Control usage problems - usage of improper keys, confusion regarding what

function keys perform● Exercise the application under controlled test conditions with users under target

audience. Data collected will be used to determine how effective, efficient, and well-received the user interface is as well as the player interactions

● Establish fundamental user performance and user-satisfaction of the user interface for future usability tests

Roles

Tester● Ask impression questions.● Observe the participant during testing and note all relevant feedback.● Ask questions that help validate the participant’s feedback.

Participant● Play the game in its entirety.● Actively provide feedback throughout the test. ● Describe any Usability issues in detail.

Usability Goals

Completion Rate

Level 1Rachel: 100%Chris: 100%Kris: 100%Dan: 100%

Average: 100%

Level 2

Rachel: 100%

Chris: 100%Kris: 100%Dan: 100%

Average: 100%

Level 3

Rachel: 100%Chris: 100%Kris: 100%Dan: 100%

Average: 100%

Level 4

Rachel: 100%Chris: 100%Kris: 100%Dan: 100%

Average: 100%

Level 5

Rachel: 70%Chris: 70%Kris: 100%Dan: 100%

Average: 85%

Level 6

Rachel: 100%Chris: 100%Kris: 100%Dan: 100%

Average: 100%

Level 7,8,9 (Broken levels, No data)

Rachel: 0%Chris: 0%Kris: 0%Dan: 0%

Average:0%

Error Free Rate

Rachel: 0%Chris: 0%Kris: 0%Dan: 0%

Average: 0%

Time On Task

Rachel: 25:42 Chris: 29:57Kris: 40:20Dan: 30:44

Average: 31:21

Death Counter

Rachel: 33Chris: 12Kris: 73Dan: 32

Average: 38

Questions

Impression Questions

Paula

1. What about the initial menu screens stands out to you the most, and how does that shape your expectations for the game?

● Rachel: I like that it’s a playable menu screen, it makes the game stand out. I think the game will be about puzzle solving. It seems like a puzzle game, requiring the opening multiple areas.

● Chris: The select menu reminds me a lot of Crash Bandicoot, where you had a room that connected to all the levels of the game. I expected a platform-esque game or some kind of call back to older games.

● Kris: To me there doesn’t look like there is a menu.

● Dan: What stands out about the menu is that you have to physically navigate the screen. It reminds me of a PS1 game.

Alex

2. What makes this game different from the others in your opinion?

● Rachel: I like the design of the Menu level. The rooftop and non-symmetry is cool looking.

● Chris: The inclusion of a ‘menu’ in the form of an explorable room as well as the main character. Main characters tend to be bipedal or humanoid, it’s kind of different to be using a ‘ghost’ as the player’s avatar.

● Kris: I am not sure what other game I am comparing it to.

● Dan: This is the first game that I played that was still in development. It is interesting, but irritating at the same time.

Lauren

3. When the game starts was there anything appealing to you that made you want to play or not play the game?

● Rachel: The way that the character looks. The Menu portals make me think there will be fantasy elements.

● Chris: The character looks interesting and with so many options in the menu room it looks like the game has a good amount of content to see.

● Kris: There is nothing that makes me want to play or not play the game.

● Dan: Nothing really stood out.

Matt

4. Based on the title alone, what do you think the game is about?

● Rachel: It sounds like an adventure game.

● Chris: Something pertaining to utilizing crystals to achieve something. Maybe something like Bejeweled or how in Mario Galaxy you had to destroy crystals to find those star bits.

● Kris: I didn’t know what the title of the game was by looking at it.

● Dan: A character with crystal limbs.

Exploratory Questions

Paula

1. Were you able to figure when to switch dimensions based on the visual cues? If so, what made it clear? If not, what would have made it clear?

● Rachel: Yes, based on the partially visible platforms.

● Chris: Just sitting for a moment and inspecting what was ahead of me made it pretty clear that I would need to use the switch ability.

● Kris: I think so. The ghostly waverly pieces indicated there was something there in a different dimension.

● Dan: Yes, I knew when to switch because on the color and transparency of the platforms.

2. What situations do you feel that the dash ability can help you through?

● Rachel: Just about everything. It saved me from death accidently sometimes. Large gaps, prolonging jumps, getting onto moving platforms.

● Chris: The situations the require me reaching a platform that wouldn’t be reached if I just jumped.

● Kris: Covering large distances, jumping out of the world, getting past obstacles created by glitches.

● Dan: Long jumps.

Alex

3. How easy was it for you to Switch Dimensions when needed? Were there specific obstacles that were harder than others?

● Rachel: Relatively easy. Moving platforms that had to be switched while jumping and dashing were much harder than others.

● Chris: Switching was really responsive, I didn’t feel like there was any delay that would make the game harder. more frustrating.

● Kris: Very easy, it was a simple press of a single key. Yes.

● Dan: The idea of switching dimensions is easy, but jumping and switching dimensions to land on platforms is hard. You really have to think about what you’re doing before you do it.

4. Were the objectives clear? Was there ever a time you weren’t sure where to go?

● Rachel: Yes, I was unsure where to go when the game broke in level 5.

● Chris: There were times when I didn’t know where to find the last crystal needed to proceed to the next level or sometimes a platform was farther than what I could reach by dashing and I didn’t know I was supposed to go another route.

● Kris: I made assumptions about picking up any glowing shard, which seems to indicate an obvious objective, although it was never outright conveyed that I noticed. Not really.

● Dan: In Level 5 I didn’t know where to go. I fell down and had no way of getting back up. I had to quit and restart the level.

Lauren

5. Did you know how to control your player at the start screen? If not, how did you figure out how to control the player?

● Rachel: No, I figured out the controls by pressing random buttons.

● Chris: I did, mainly because most platformers use the WASD control scheme.

● Kris: No, but my initial guess was correct.

● Dan: I used the arrow keys to control at the menu.

6. At any point did you think there should be further explanation about how something worked? If so, when did you think you needed more explanation and with what feature?

● Rachel: No.

● Chris: The dash bar was a bit confusing, just because it seemed like I couldn’t dash unless it was at 100% but it only took away 50% of the bar so it felt like I could do an additional dash

● Kris: Much of the explanation text did not remain on screen long enough to read, which is a problem unto itself, but also means I can’t be certain of my answer to this question as I’m not sure of what all was contained within the explanatory text.

● Dan: Switches, it is not easy to tell when they are activated, and what their effect was. I also had to figure out that the Dash had a cooldown.

Matt

7. Did you find the powerups useful? Which was the best one and why? Which was the worst and why?

● Rachel: Yes, the dash was necessary for completing every level. I never found any other power ups

● Chris: The dash was extremely useful, it felt more intuitive than the other power-ups. The dash allowed me to explore the map more in some ways. Like I could reach the land’s edge next to the lava where I wouldn’t usually be able to go.

● Kris: Yes. Dash because it enabled me to get around many obstacles, even ones that didn’t seem intentional as well as take short cuts. Dash, because I only received one power-up, therefore it must be the best and worst.

● Dan: Dash was the only power up that I got. It was very useful for long jumps and navigation, and obviously the game couldn’t be completed without it.

8. Does this game compare to any you have played before, whether it is aesthetics or gameplay.

● Rachel: The moving platforms remind me of Mario

● Chris: The older games like Mario 64 and Banjo Kazooie come to mind. Pretty much your standard, open-world platformer!

● Kris: This game reminds me off A Link to the past. It reminds me of this because of the two different worlds that you switch between.

● Dan: It reminds me of PS1 platformers like Spyro, and of Ratchet and Clank with the moving platforms.

Initial Testing Experience

Throughout the playtesting experience there was a variety of issues that blocked the progress of the testers.

There were plenty of bugs and issues that had the character either falling through the world. At this point the only option for the player would be to exit the level and restart it all over again.

Other bugs were present that either locked the movement of the camera or the movement of the character. Playtester Rachel ran into an issue where she could only jump, but not move the character forward or back. Playtester Dan also ran into a similar problem at the beginning of level 5, where the character’s movement and camera was locked.

Problems also arose where certain design aspects didn’t perform their functions. As seen in the picture below, there were multiple lava pools in the game where the player could stand atop and not die.

The in-level instructions also halted some playtesters progress. When collecting the Dash power-up, players are shown written instructions on how to use it. Playtester Dan didn’t have enough time to read these instructions, so he was unaware of the controls.

A poor checkpoint system led to the frustration of the playtesters. There are no checkpoints at all present in Level 5, and very few throughout Level 3, which led to the players being brought back to the beginning of the level. Having to replay the same parts over and over again will lead the players not wanting to continue.

The Dash Power-up showed the presence of level design that wasn’t ideal. Playtester Kris found himself being able to skip past obstacles and platforms by using the Dash Power-Up. This defeats the purpose of the some of the level design choices and makes the game too easy for players. The Dash ability also allowed the character to fly above the entire world, being able to see the framework of the level. Throughout some levels there are air currents that launch the player upward, and when the Dash is used while on one of these currents, the player soars upward, like in the picture below.

A unique issue that one of our testers dealt with came from the level select. To unlock

subsequent levels players must collect shards in the game. In order to unlock the second level, the players must have 10 shards, 20 for the third, 30 for the fourth, and so on. The problem lies in the fact that there are more than 10 shards in the levels. So it is entirely possible to collect enough shards to completely bypass a level. For instance, Playtester Kris had enough shards to skip level 6 and go straight to level 7.

Flowchart

References

Usability Test Plan Template. (n.d.). Retrieved December 11, 2014, from http://www.usability.gov/how-to-and-tools/resources/templates/usability-test-plan-template.html

Crystalim. (2014). Team Crystalim. Winter Park, FL: Full Sail University.