Category Archives: Visual

A Better Gamepad Keyboard: Part 2, Slashcard’s Dual-Stick Input

(This article is also available as a devblog on Gamasutra.)

Slashcards: Learn Japanese! (other languages forthcoming) is a local co-op language-learning action game. This is a game that asks gamepad-using players to respond to free response language prompts (among other types of prompts) — and that demands a faster input method then the onscreen keyboard you configure a PS4 with.

This design blog follows part 1, an exploration of extant gamepad text input designs.

After researching extant gamepad text input methods, I found that none of those options would work to type in even a few letters in a real-time action game situation. (Moreover, I’m not counting on accelerometers or IR sensors for any Wii or PS4 fanciness.) If Slashcards is to be workable as an action game, I need a better solution. I was sure I’d at least need a dual-stick input method, so I began with the design described in the Microsoft paper.

Iteration 0: +Dual-stick, hunt-and-peck, divided keyboard

Dual stick keyboard

Dual stick input seemed a natural extension from single hunt-and-peck keyboard input, and I was glad that someone had given it a shot over the past thirty years. But I was disappointed to see that their results showed such a modest increase in input speed.

Mean WPM from Microsoft study

This design consists of dividing the keyboard in two and putting the cursors where your respective middle fingers would rest on the keyboard — the left cursor defaulting to “d” and the right to “k”. To type “q” would mean pressing left, left, up on the left stick and then a left shoulder button. “m” would be down and then left on the right stick, and then pressing a right shoulder button.

Like the single-cursor, hunt-and-peck keyboard, cursor positions are persistent. So once the left cursor is on “q”, it takes seven inputs to type “b” (left-stick-right, right, down, right, right, down, left-shoulder.)

Iteration 1: +Elastic cursor

I played with having the cursor positions reset after an input. On the one hand, this behavior is less obvious than leaving the cursor where you left it. But in exchange for a slightly steeper learning curve, this choice provides the user with an opportunity to develop muscle memory for every letter. “w” is then left-stick-up, left-stick-left, left-shoulder every time, regardless of the previous letter. After a few minutes of use, I could feel myself getting faster. But typing whole words still felt like a tedious amount of input. If every letter is two to five inputs, a five letter word ends up being around sixteen inputs.

When I tried mocking it up myself, I found that my own performance was already a bit better than that of Microsoft Research’s test group. But the hunt-and-peck approach with its concomitant repeated directional inputs — even with the keyboard divided — won’t reliably work in any kind of time-constrained, action game context.

Iteration 2: +Free selection (non-hunt-and-peck)

Wouldn’t it be nice if I could choose a letter with a single stick-movement? Dispensing with the stateful cursor position, I yoked cursor position directly to the stick position.

Under this system, on the left stick, leaving the stick idle would select “d”, pressing all the way to the left would select “a”, pressing to the upper-right would select “t”. The cardinal directions and their diagonals are easy enough…

Naive stick->keyboard mapping

…but that leaves the intermediate letters to be related to somewhat tricky intermediate positions.

Iteration 3: +Optimized free selection

The difficulty here can largely be addressed by optimizing the mapping between stick and virtual keyboard. The guiding principle is to give each option as much selection space as possible on the stick. So if we map out the space on the stick that can be selected, we can see that the original, naive mapping is obviously, needlessly difficult:

Naive mapping (perception)

And while it’s tempting to contrive a mapping where each option literally has equal area on this mapping, like above, it ignores obvious and massive optimizations the controller gives in context. For one, the neutral position needs zero area — the stick reliably snaps back to (0.00, 0.00). The cardinal directions are also essentially a zero-thickness line, wherein each has a coordinate of 1.00 (north, or “e” on the left stick, always has y = 1.00.)

Naive mapping with minimized central sector

In practice, I found that the other coordinate of the cardinal directions was always very close to 0.00. And indeed, all the keys around the edge of the selection area (QWERT-G-BVCXZ-A, to go around clockwise) had a distance from the center that was reliably greater than 0.9.

Improved mapping

Testing and experimentation gave me the final values I used to divide up the mapping, and the end result looks something like this:

Perceptually optimal mapping

The result is a massively improved input rate–provided the user is willing to put up with an initial learning curve of this new interaction. I encountered testers who, having spent countless hours hunting-and-pecking over Xbox Live and whatnot, were initially quite frustrated by this system. But even in their ire, they were nevertheless actually typing faster than they had been.

Free selection demo

Iteration 4: +Quadrants

Far and away the most common errors were in the corners, such as NNE and NE. My anecdotal suspicion is that gamers are used to gross input, be it little taps to line up sniper sights or the directional inputs for fighting game power-moves. Therefore I wanted to accommodate an input gesture less precise but more reliably repeatable.

My solution for this was to divide each split-keyboard half into quadrants.

The user would first choose a quadrant by pressing in one of the cardinal directions and then (optionally) turn to another direction to select the key within the quadrant.

Pressing right on the left stick would select the “FTGB” quadrant, and moving up would select “T”; moving down, “B”, pressing all the way to the right would select “G”, and a stick position not on the right edge of the joystick would be “F”.

Free selection with quadrants demo

Therefore typing “B” would reliably be one motion consisting of flicking the left stick right, then down. In practice, that basically feels like rubbing just below the center of the right side of the stick edge, and it’s endlessly reliably repeatable.

typing with the quadrant keyboard

Fighting the quadrants can be frustrating, however. If you accidentally select the right quadrant and want to get down to the bottom, sliding down from the right edge won’t work; the stick needs to be relaxed, or the cursor needs to otherwise return to the center first. Therefore some users, whose grip on the gamepad sticks is lighter and more precise, will prefer iteration 3.

That said, my limited test group performed best on the quadrant keyboard. And here’s the split quadrant elastic dual stick on-screen keyboard (SQuEDSOSK) in context:

Slashcards typing in game

(You can see how I’ve taken steps to keep the on-screen keyboards from occluding too much — more work to be done, there, to be sure.)

Try it out!

If you’d like to try Slashcards: Learn Japanese, you can give the pre-release preview a spin by downloading a build from this page:

Why stop at English? Keep an eye out for part 3, where I’ll talk about the development of the Japanese-specific kana keyboard.

(TIGsource forum post version)

A Better Gamepad Keyboard: Part 1, A Survey of Extant Gamepad Keyboards

(This article is also available as a devblog on Gamasutra.)

Slashcards: Learn Japanese! Gameplay

Slashcards is a game about learning language. And part of rehearsing new language knowledge is responding with English equivalents of pronunciation. The most thorough way of testing players is for them to input that pronunciation letter by letter. So, one interface challenge I embraced for Slashcards’ design was to make a game-friendly gamepad-compatible onscreen keyboard.

Part 1: A survey of extant gamepad typing interfaces.

In an effort to avoid re-inventing the wheel — and falling into old ruts — I did some research on to typing on consoles. Perhaps the most naive approach of all was to make a matrix of buttons where each button corresponded to a key on the keyboard. The user would then navigate through the matrix like any kind of on-screen GUI, going from GUI element to GUI element by pressing the d-pad or left stick in the corresponding direction. This has remained ubiquitous ever since 8-bit consoles. Here’s The Legend of Zelda (NES, 1986):

The Legend of Zelda

…and here’s the Xbox 360, 20 years layer: (around its launch, ~2006)

XBox 360 ABC Keyboard

The Xbox 360 keyboard originally defaulted to an ABC layout. I guess ABC-order is defensible insofar as we all know the alphabet, so, given a letter, we would intuit the neighboring left and right letters. But what about the letters above and below? It’s frustrating that, for example, P is to the left of L. Moreover using the wrap-around topology of the keyboard — pressing left on the left edge to go to the right edge, and vice versa — would similarly be a convenience only available to those who carefully studied this unnatural layout.

Eventually — perhaps from release, I don’t recall — you could change the layout of the Xbox 360 on-screen keyboard to a QWERTY layout, and leverage the familiarity every modern person has from thousands of hours of typing. Now you could hunt-and-peck like you were typing on your laptop as you would while you ate cereal or nudged some stir-frying onions with the other hand. I believe this is now the default layout on the Xbox One.

Xbox Keyboard QWERTY

Sony has offered a bizarre multi-step interface, whereby users hunt-and-peck a letter family on the left. Then a number of options including auto-complete suggestions are presented on the right. Finally a user selects one of those options by again navigating across them. The result is a mixed bag. Letters that were far from each other in the above keyboards are closer in terms of button press-count, but previously-adjacent letters might be five or six button-presses in the PSP system. Maybe this compromise plus the autocomplete suggestions turns out to give more words per minute — I remember my time typing on the PSP as an exercise in frustration where I had to relearn the system every time I encountered it. Either way, the bottom line is that any design that relies on autocomplete suggestions is not going to be appropriate for Slashcards.

PSP keyboard

Fortunately for PSP fans (and the PSP’s successor, the Vita) eventually the PSP also offerred a full keyboard layout (known as the “fullscreen keyboard” on the PSP.

A far better performing option is the approach taken by the Wii and the PS4 on-screen keyboards. They offer a QWERTY keyboard whereby the player can more or less point to a key to select it. (The PS4 requires players to tilt, not exactly point, but the result feels responsive and intuitive.)

PS4 accelerometer keyboard

Steam Big Picture mode has an interesting hierarchical keyboard. The user presses a direction with the left stick and selects one of the four action buttons to select a character.

Steam Big Picture Keyboard

The more you use this keyboard, the faster you’ll type — and the skill ceiling is far higher than the hunt-and-peck keyboards above. It also has the virtue of being alphabetical but without the compromise of arbitrary rows that computer-keyboard-lookalike layouts have. At every letter you can easily see if your next layer is 1) in the current button group, 2) counter-clockwise (previous to the current letter) or 3) clockwise (after the current letter.) This is the first system that really tempted me towards implementation in Slashcards. I struggled mightily to compress it to a size that wouldn’t be so demanding of screen real estate. I couldn’t come up with a workable solution but you’ll see that it has something in common with the Japanese input solution I devised.

My first thought was to take the extant QWERTY hunt-and-peck keyboard that we see on the Xbox/PS and add another cursor. Players could hunt-and-peck with the left stick and right stick. This seemed obvious enough that I wondered if this had been attempted before — sure enough, a Google search hit showed me that the approach is explored to some extent in a Microsoft research paper.

Their research finds study participants went from 5.8 words per minute on the single-stick hunt-and-peck QWERTY keyboard to 6.4 words per minute with a dual-stick split hunt-and-peck QWERTY keyboard. A 10% gain is still far from the kind of quantum leap I’m looking for.

In the end, I found that none of these options would work to type in even a few letters in a real-time action game situation. Slashcards demands better!–or rather, a more precisely-designed solution.

In the next part, I’ll show how I mixed and matched the best of the above approaches to iterate towards a functional, consistent, and radically more efficient on-screen gamepad keyboard.

Slashcards Unity Quick Tip 2: Bringing a face to life

This is a continuation of Quick Tip 1, which discusses the coding approach to the faces in Slashcards: Learn Japanese.

Character still.

Let’s bring our character to life.

Plop her in your fantasy world and what would she be doing? She’d be looking around, thinking and reacting to what she saw. That means she’s blinking. She’s raising her eyebrows or furrowing them as she thinks about this or that. She’s pursing her lips and relaxing them. And none of this is happening in a rigid, repeating pattern. It’s all semi-random. Sometimes she blinks three times in three seconds. Sometimes once in ten seconds. Something will worry her for just the briefest moment…etc.

For my character, I’ve broken facial activity into four processes:

  • Blinking
  • Looking-at
  • Eyebrow movement/shape
  • Eye shape

Moreover, the character has a Mood and a PointOfInterest.

Blinking is a coroutine that blinks every so often (from .5 to 5 seconds). It’ll blink more often if the mood is Afraid or Sad.

LookingAt is a coroutine that aims the pupils at the PointOfInterest if there is one. Otherwise it just picks a different thing/point in space to gaze at every several seconds.

Eyebrow movement/shape shifts the eyebrows up and down at random intervals. The eyebrows’ shapes are determined by the mood.

And eye shape — angry, relaxed, afraid, and so on…

Eye shapes by deviantartist sharkie19.  You can imagine how you might break out eyebrow, pupil, and eye shape.

…are controlled by a coroutine that varies them according to the mood. Certain moods use different sets of eyeshapes. For example, in Mood.Relaxed, the character might momentarily be pensive/worried — an eyeshape also used in Mood.Afraid — but it won’t use that expression as often.

Slashcards facial animation

Crucially, this approach shows the player that the character has an inner life. Keeping the control in code makes it easier to vary. And the varied behavior means the designer/animator doesn’t have to handcraft every little reaction.

Slashcards Unity Quick Tip 1: Fast Faces

One thing I wanted to achieve in Slashcards (check out the playable preview at itch) was characters that had some character. I wanted them to feel alive. One game that does that superbly well is The Legend of Zelda: Wind Waker.

Because of the specific artistic choices in that game, the characters faces were basically drawn on as if cartoons. That permitted a ton of expression with a bit less work than, say, modeling a humanoid face, rigging it, and using motion capture or hand-crafted animation for each little expression.

To make expressive faces, you first and foremost need eyes and eyebrows. Eye shape and eyebrow position tell you just about everything you need to know about someone’s emotional state. I wanted to be able to control the eye pupil position by script, so rather than have canned images fr looking left, looking right, etc., I chose to have the pupil be a separate image from the white of the eye. And that meant masking the eye, which means shaders, which means more draw calls…

Two eye brows, two eyes with two pupils, and we’re already up to six draw calls per character, and we haven’t even gotten to the mouth, yet. The trick to keeping this cheap is (in Unity) using MaterialPropertyBlocks.

The drawback of MPBs is that you have to set them in code. And if you want to see the results in the editor, you’ll need to have a script that sets their values in OnValidate.

So the recipe is: use the same texture, use the same shader, and to vary parameters across different facial elements, use MateriapPropertyBlocks. Next up, I’ll get into how they’re animated.

looks painful..
Hm, looks painful…