Directory of All Essays

Saturday, December 06, 2008

Post-it note design docs


I happen to fall into the artist-designer skill set, so I often find myself trying to prototype ideas on teams rich with programmers. As such, I'm always looking for better game development techniques that work well for this particular team mix.

Here is a very lightweight prototyping process using Post-it notes that I quite enjoy.

  1. Initial idea: I sit down with an available programmer (and artist/UI designer depending on the system) and we chat about how to test out a new bit of gameplay. Usually this is an idea that has been bubbling about since the night or week before.
  2. Post-it note design: I jot down a quick bulleted list summarizing our discussion on a single post-it note. We go over it one last time so there everything is clear. The list isn't very detailed. Mostly it serves as something to jog our memories if we forget what we were talking about.
  3. Build it: The programmer and artist go off and build the items on the list. It might take 2 hours or two days. They are encouraged to solve problems creatively and they can always just give me a shout if something doesn't make sense.
  4. Play test: When most of the items on the Post-it note are playable, I get called over and we play test it the experiment together. If the results are comprehensible by mere humans, we pull in some play testers for 3-4 minutes to observe some real players interacting with the mechanic for the first time.
  5. Review: Afterwards, we discuss our observations and write up another Post-it note worth of improvements.
  6. Repeat or Stop?: The process repeats until we run out of time or give up. Sometimes we give ourselves a day per experiment, sometimes two days. In the land of Scrum, we treat the experiment like a time boxed task.
  7. Rate: At the end, the gameplay experiment is rated according the scale below.
  8. Save: The code is saved off, a few choice notes are recorded in a doc containing our 'list of experiments' and we move on. Bits of code, even from failed prototypes, are often reused in future gameplay experiments.
Rating system
The rating system is delightfully crude. The goal is to triage experiments quickly.
  1. "A": These experiments were obviously fun. Players laughed, smiled and generally exhibited the emotions we were looking for. If in doubt, ask "Was this fun? How so?"
  2. "B": These experiments showed a hint of fun if you knew what you were looking for. However, it is going to take more effort to expose the fun in a manner that is visible to the players.
  3. "C": There wasn't any fun. The experiment fails.
A portfolio of fun
One of my favorite aspects of this method is that you end up with a mini-portfolio of game design ideas. Instead of putting all the design risk in a project on one or two unproven mechanic, the team now has a half dozen or more proven bits of fun to build upon. If some don't fit into the game or get abandoned for other reasons, that's alright. You can afford to lose a few and the end product will still be fun. Think of it as designing from a position of plenty.

Contrast this with a prescriptive 'design doc' approach where you are forced to pick, without much evidence, a main mechanics for production. Even for the most experienced designer, 50% to 80% of your 'educated' selections are going to be complete dogs. Every unproven mechanic you polish ends up being a massive drain on your budget and your reputation as a designer. You might hear gentle comments like, "We spent 3 months of dev time on this lump of an idea and it isn't fun?"

It doesn't take very many expensive failures for the project's perceived 'design risk' to blossom to the point where conservative minds seek to kill the project. I think of this as designing from a position of sudden death.

Some basic observations
Here's a quick list of things I've observed when prototyping.
  • Failed experiments happen a lot. Don't be surprised if C-rated experiments occur 50% to 80% of the time. Everyone on the team has to be aware that not every experiment is going to be a success, but the learning process is still worthwhile.
  • Designing on your feet is a critical skill: Each consultation and analysis might last only 10 to 20 minutes and you need to leave folks with that all important sticky note filled with impactful, yet inexpensive changes. It pays to have lots of ideas and a deep understanding of game mechanics so you can quickly pull together a list of incisive comments. If you can't, you likely are not suited to be performing the designer role.
  • Listening matters. The designer doesn't need to come up with all the solutions. Everyone on the team is bright and has great ideas. As a designer, your role is to herd all ideas (yours and others) into something that serves the next step in the prototype.
  • You need programmers: If there aren't programmers dedicated to prototyping, the prototyping isn't going to happen. You can drop down to paper prototyping, but it usually doesn't prove out many types of mechanics (especially ones involving timing and interfaces.)
Advanced observations
These are some notes that are a bit geekier, but can save you large amounts of pain.
  • Meta game mechanics are harder to prototype: The systems that link together the various gameplay experiments are harder to playtest. They operate on longer time spans (hours instead of minutes) and often require that the core gameplay is already fun.
  • Build a meta-game architecture that allows for loose coupling of gameplay experiments: Most successful games have an architecture that allows the team to plug in new bits of fun as they are found. The linear 'level-story-level' pattern used by most FPS is one example. The 'hub with many sub levels" used by Mario 64 is another. Any of these allow you to plug in a new experiment independently of the other gameplay experiments. If you don't have a modular architecture, you run into situations where a fun new system breaks many of the other bits of fun you've already discovered.
  • Integrating tightly coupled gameplay experiments is a pain: If I independently find a fun new type of weapon and an interesting enemy AI, the combination of the two is often a non-trivial issue. The new weapon many work with an old AI, but be completely useless with the new one. Integration yields a whole new set of experiments. Plan for time to rediscover the fun all over again.
Benefits
There are some interesting benefits to the Post-it note design method:
  • Scales nicely to large prototyping efforts: One designer can serve multiple programmers. This works nicely on teams where there are more programmers than designers and you need to get a lot of prototyping done quickly.
  • Failing quickly is fun and educational. You learn a lot with each failure and can move onto the next experiment with a much better idea of what doesn't work.
  • Provides a quick death for bad pet ideas. It is much harder to resurrect pet ideas when you have concrete, playable proof that it won't work. Finding out early which one of my favorite ideas is idiotic saves me a lot of political pain.
  • Fun prototypes are quite convincing: A fun, playable crazy idea works a lot better for winning over other team members than any amount of hand waving or documentation.
  • An easier team to assemble: Finding a competent game designer and a competent programmer can often be easier than finding a competent programmer-designer. Well developed hybrid skill sets are very valuable, but can be quite rare. A side benefit of having a team is that you end up cross training your designers and programmers. You create designers who can speak to programmers and programmers who can riff on some of the design.
The value of dime-a-dozen designs (A brief aside)
One often hears the negative comment that game designs are a dime-a-dozen. And in a waterfall design process, an incessant stream of ideas is indeed a problem. If you attempt to squeeze all those ideas into a typical waterfall development process, you end up with an immense amount of waste. Each designs need documentation, concepting, implementation, testing and bug fixes. In response, project owners will often ask for just one good idea.

There is another path. A lightweight prototyping method takes your flurry of crazy ideas and converts them at moderate cost into a well sorted portfolio of working designs. All those ideas are not, in fact, worthless or wasteful; they are the essential fuel that feeds a good prototyping process. Each idea either teaches you something or provides you with a success.

The way to make the process work without getting gunked up is to make prototyping incredibly lightweight. Other than our focused conversations, I spend my time on a total of two design docs: The first is the brief list of rated prototypes and the second is a set of discardable, temporary Post-it notes. Design waste in the form of unnecessary artifacts is minimal. Most of the 'programming waste' is better classified as low cost learning.

Those wild flocks of churning, swirling ideas end up not being worthless at all. They simply need to be funneled into the project with the right process for their value to be realized.


Conclusion
The "Post-it note design process" has likely been reinvented in one form or another hundreds of times across the history of game development. It is so basic that it feels odd to even write it up in any sort of formal fashion.

If you have a designer and a programmer, give it a shot. It is certainly a good functional alternative to the popular process of sticking a lone programmer-designer in a room and asking them to 'find the fun'. Both can produce great games. Pick the one that works best for your current team composition.

This process does have an cost since you need to devote at least two people to finding the fun instead of putting all decisions on the head of the designer. However, the end result is well worth it. After all, it is far smarter to spend team time uncovering a portfolio of the right mechanics than it is to 'save your programmers' so they can be off running really fast in the wrong direction.

In the end it really isn't about programmers, designers, design documents or features. It is about the team working together to make the right product. Everything else is just ego and waste. And for some reason, it is quite difficult to invest much ego or waste in a little disposable Post-it note.

take care
Danc,
Post-it note fanboy

Labels: , , ,


Read more!

Monday, May 19, 2008

Improving Bug Triage with User Pain



The traditional bug triage process is miserably inefficient. Over my decade in this industry, I’ve spent months of my life sitting in windowless offices manually reviewing (and re-reviewing) thousands of bugs. Often times, there are three or four folks on the triage team, typically the most skilled people on the team, sitting about and bickering for hours over the finer points of obscure bugs. Politics, boredom and arbitrary decisions are unfortunately common. The result is wasted time and poorly managed bugs.

So we came up with a better way.

User Pain is a technique I’ve been using for many releases across multiple teams. It involves sorting bugs on a single unified scale called User Pain that takes into account common bug ranking criteria. I’ve found that it can reduce the cost of triage, help teams ship on time and greatly clarify which bugs you should be fixing right now. This essay describes how User Pain works and some best practices for implementing it on your team.


Problems with current bug triage
Traditional bug triaging is a time consuming and tedious process. Bugs come into a bug database with little prioritization, the team leads sort and rate each problem and then assign them to the appropriate members of the team. This process tends to run into several issues:


  • Lack of shared criteria: Different people often value different aspects of a bug, which leads to unhealthy disagreement. A designer might think a usability issue is a critical fix, while a programmer might be concerned about a crash. With no common criteria, it is hard to build consensus quickly.
  • Wasted time: Often the highest skilled team members are required to triage bugs. They spend hundreds of hours poring over mundane issues again and again. This is time that could be better spent improving the product.
  • Bottlenecks: Bugs are often required to go through a review process so that precious developer time isn’t spent on bugs that would have otherwise been punted. The loop from the submitter to the triage team to the developer can cause delays for critical bugs.
  • Big undifferentiated bins of bugs: Since the incoming rate of bugs is often higher than the fix rate, large piles of bugs will accumulate for each developer. If a developer has 50 bugs on their plate, they will fix them in a semi-random order or rely on micromanagement by the triage team. The first tactic means critical bugs are sometimes left to be fixed until the end. The second means more time is wasted on reviewing bugs.
  • Triage burn out: After reviewing thousands of bugs, many triage teams stop caring or become fixated on a few bugs at the cost of reviewing others. The result is that some bugs are poorly triaged and the quality of bug ranking in the database is low.
These are the problems we want to solve with User Pain.

The basic system
User Pain is yet another technique inspirted by the world of Lean Manufacturing, the ancient mother of so many Agile practices. The technique was original developed in the 80’s as a method of efficiently classifying product defects on manufacturing lines. While some of the ideas are new to software development, the core concepts have been tested in intense product development situations for decades.

At with many agile techniques, User Pain isn’t all the complicated.
  1. Rank each bug on several criteria
  2. Combine those criteria into a single score called User Pain
  3. Sort all bugs by User Pain into a public list
  4. Start fixing the most painful bugs at the top of the list.
There is a distinct philosophy at work here. First, empower bug submitters to easily create well formed, well classified bugs. Next, give the team the tools and information necessary to make smart decisions about what to work on first. Finally, encourage practices that make it easy to put quality first. Instead of relying on expert managers, you rely on a well informed, empowered team. As a result, the User Pain system removes most of the need for a triage middleman.

Let’s dig into each step and explore the devil in the details.

Step 1: Rank each bug on several criteria
Bug submitters use a simplified bug submission page that clearly lists three factors: Type, Likelihood and Priority. Each factor has multiple values, listed in order of impact. At submission time, the bug submitter rates the bug.

Three bug rating factors
Here are the three factors that I’ve been using.
  • Type: What type of bug is this? For example is it a crashing issue, a problem with localization or a matter of visual polish?
  • Likelihood: How likely are users to experience the bug? For example, does everyone run into the issue or do only a few users run into it?
  • Priority: Of the people who experience the bug, how badly does it affect their experience with the product?
These particular factors have been battle tested for many a release and were selected for the following reasons.
  • Good coverage: These cover the range of concerns expressed by most stakeholders. Type includes business priorities while Likelihood and Priority help classify user impact.
  • No overlap (aka orthogonal): A bug can be rated on one factor without affecting how you would rate the other factors. This allows you to rate each factor in isolation and greatly improves the objectivity of the results.
  • Small number: There are few enough of them that they don’t overload the bug submitter. It is easy to add more factors for various edge cases, but typically this results in a cluttered and confusing bug submission form.
Use anchored scales
Now that we have our factors, each one needs a rating scale. At this point, we do something slightly tricky. Each point on the three scales is anchored to an objective description. Here are the anchored scales I prefer:

Type (What type of bug is this?)
  • 7: Crash: Bug causes crash or data loss. Asserts in the Debug release.
  • 6: Major usability: Impairs usability in key scenarios.
  • 5: Minor usability: Impairs usability in secondary scenarios.
  • 4: Balancing: Enables degenerate usage strategies that harm the experience.
  • 3: Visual and Sound Polish: Aesthetic issues
  • 2: Localization:
  • 1: Documentation: A documentation issue
Priority (How will those affected feel about the bug?)
  • 5: Blocking further progress on the daily build.
  • 4: A User would return the product. Cannot RTM. The Team would hold the release for this bug.
  • 3: A User would likely not purchase the product. Will show up in review. Clearly a noticeable issue.
  • 2: A Pain – users won’t like this once they notice it. A moderate number of users won’t buy.
  • 1: Nuisance – not a big deal but noticeable. Extremely unlikely to affect sales.
Likelihood (Who will be affected by this bug)
  • 5: Will affect all users.
  • 4: Will affect most users.
  • 3: Will affect average number of users.
  • 2: Will only affect a few users.
  • 1: Will affect almost no one.
An anchored scale describes each point on the scale with specific, objective criteria. As long as the item being rated meets the criteria, you know what value it should be assigned. An anchored scale is preferred over a relative scale (ex: Please rate this problem from 1 to 10) since there is less subjective judgment involved in assigning the value.

Display the anchored scales prominently on the UI where bugs are entered
Anchored scales are only useful if the team can see the descriptions.

On one team, we only displayed values 1 to 5 in a drop down list and asked submitters to remember what each value meant. This wasn’t very effective. People treated each factor as a relative scale and would rate items by ‘feel’ initially instead of referring to the definitions of each value. The end result was that bug ratings were heavily influenced by personal preference.

Instead, build a bug submission UI that lets the submitter read the descriptions as they rate the bug. Radio buttons work wonderfully since you can place all the descriptions right in front of the user. A drop down that contains the descriptions is also feasible.

This may seems like a minor issue, but people are usually in a hurry. If you don’t make the rating process painless, they’ll happily toss random data into your bug database. Improving the clarity of your bug submission UI is the single cheapest thing you can do to improve the quality of your bugs.

Who enters bugs
This system is intended to be used by members of the development team. Artists, testers, developers, designers, project managers and producers all should be able to understand the criteria and enter well rated bugs. They'll need an understanding of the core scenarios and the target user. The better that you educate the team on what you are doing and who you are doing it for, the better your bugs will be.

This system does not work well for bug submissions by external users. They don’t understand the terminology and tend to create bugs that are poorly formed. A good solution is for a tester to reenter the user bugs with the proper ratings and format. It is a form of triage, but is a relatively minor cost in the grand scheme of things.

Benefits
Using anchored scaled for rating bugs upon submission has the following benefits.
  • Less reliance on personal opinion: A tester who has some domain knowledge can quickly classify the bug into one of the buckets without relying overly much on their personal opinion. The result is that even when multiple people independently rate the same bug, the final user pain tends to cluster very tightly around the same values.
  • Harder to game the system: Anchored scales also make it harder to simply ‘bump the pain’ up for a bug that has become a hot topic. Due to the clarity of the rating categories, poorly rated bugs are usually flagged by the next person who looks at them.
  • Push triage to the submitter: When you can trust the ratings set by bug submitters, you can eliminate a large portion of the triage process. Provided that your submitters have basic domain expertise, 80-90% of the values that they set during the initial submission stay the same throughout the life of the bug. This means that there is less need for reviews to reset random values.
Step 2: Combine those criteria into a single score called user pain
Once a bug is ranked on all three factors, you multiply the numbers together to get the User Pain score. User Pain is a single value that can be used to compare widely divergent bugs. User pain deals with the gray area in which most bugs live.
  • Obvious fixes: A bug that the users hate, blocks major user scenarios and affects everyone causes a big hit to perceived product quality. So it receives a very high pain score.
  • Hard calls: A bug that occurs all the time, despite the fact that it blocks only minor scenarios, still receives a moderately high score.
  • Tricky punts: A crash that is never seen by anyone receives a low score.
The basic equation for calculating User Pain is as follows:
  • User Pain = Type * Likelihood * Priority / Max Possible Score
User pain is auto calculated when the bug is entered and whenever the bug changes. After you calculate user pain for a set of bugs, you’ll find that you have a smooth spectrum of bugs ranked from 1 to 100% Pain.

Benefits
  • One value for comparing different bugs: Instead of forcing users to juggle multiple different criteria when comparing bugs, they only need to look at one. This means quicker judgments and easier sorting.
  • Fewer big bug buckets: No longer do you need to deal with huge swathes of undifferentiated bugs. Instead of dealing with 300 priority 2 issues, you typically will see much more manageable clumps of 3 to 5 bugs with the same pain rating. Bug Maturity, as described in the Appendix also helps spread out the bugs.
Step 3: Sort all bugs by User Pain into a public list
Once you have your list of bugs complete with user pain, you need to display them to your team in an easy to understand manner. I’ve dabbled with custom queries inside bug tracking tools, but my favorite technique is to create the world’s simplest bug dashboard.

The Pain List

The Pain List is a webpage that lists all the active bugs in order of User Pain. You put the highest pain bugs at the top of the list and you make each bug a hyper link that takes you to the bug details in your bug database. Be sure to auto reload the list every 10 seconds or so the data is fresh and reliable.



The Pain List becomes your central dashboard for daily bug management. I’ve gone so far as to make it the homepage on my web browser.

Quality bars
The team can use the Pain List to set easy-to-understand quality bars as exit criteria for your milestones. For example, they can say “In order to release, we want no bugs greater than 30 pain.” At the 30 pain threshold on the Pain List, you draw a line. Anything above the line needs to be fixed. Anything below the line you can ship with.

Quality bars can be more meaningful than traditional bug counts since you are implicitly taking into account the final user experience. Meeting this bar means that you’ve fixed all crashing and unpleasant bugs and the only issues that are left are minor cosmetic ones that are rarely seen by users.

Since you have a finely incremented spectrum of bugs, you can also precisely adjust quality bars based on your place in the release cycle. You could set a high pain threshold if you are dealing with new features. You can tighten the quality bar further for subsequent releases.

Benefits
  • One view: One view shared by everyone, including both testers and developers. You don’t have to worry about juggling divergent database queries.
  • Simple to understand for all parties involved. There are no specialized tools or incomprehensible graphs. Even management can know where you are at just by glancing at the list.
  • Clear understanding of status: If there are bugs above the quality bar, you need to start fixing bugs.
Step 4: Start fixing the most painful bugs at the top of the list
Now that we have the Pain List, we can finally put it to use. Developers check the Pain List daily and fix the highest pain bugs on the list. If there are no bugs left above the current quality bar, they work on feature work. This basic heuristic is a surprisingly efficient method for managing quality.

Assigning bugs
All bugs are assigned to Active upon submission, not a particular developer. When a developer sees a high pain bug that they want to work on, they assign it to themselves. The bug then goes through the standard process of being fixed, tested, and closed by the submitter. In general, developers should have no more than a half dozen bugs assigned at once. They pop items off the list, fix them and go back for more. Hoarding is highly discouraged. So is assigning bugs based on feature area.

Fixing bugs before feature work
All bugs above the quality bar should be fixed before new feature work is started. If you follow this practice, you should exit each sprint with no more high pain bugs than you entered the sprint. Bug debt doesn’t accumulate.

This practice helps you build quality in as you develop. When this practice is paired with solid automated tests, you enter into a whole new world of high quality development.

My bugs
At the top of the Pain List is a section that lists the current user’s assigned bugs. This both helps devs treat the Pain List as their entry into the bug database and it reminds them that they should finish the items on their plate before taking on new work. Since this list is short, it rarely interferes with browsing the Pain List.

Benefits
  • Developers always know what to fix: All they need to do is look at the top of the list and there is almost always a bug waiting for them to grab. As a result, developers never need to juggle multiple criteria in their head when deciding what to work on next. Nor do they have to wait on leads or managers to assign them bugs.
  • Promotes shared code ownership: The rule ‘fix from the top’ rarely correspond with ‘fix the code that I developed’. Short term, this is less efficient since developers may need to ask questions of the original developer about an area of the code. Long term, the broad knowledge of the code base that comes from fixing bugs outside area of expertise results in higher overall productivity and team flexibility.
  • Bugs that prevent you from shipping don't accumulate: The benefits of fixing bugs before features are numerous. The pain of shipping is greatly reduced. Testing is more effective since they don’t need to constantly juggle workarounds to problems that won’t be fixed for months. Finally, the team feels better because they know they are always building a high quality product.
Pitfalls
There are a few pitfalls that emerge when you first try to implement a User Pain system.

Training the team
There are likely people on your team that have been dealing with bugs for decades. Changing the bug tracking system will require retraining before you see positive results.

In my experience, new teams initially rank 80% of the bugs incorrectly because they A) do not use the rating scale or B) do not understand the target user. To fix this issue, keep making the anchored scales highly visible and keep promoting the major scenarios and target user. After people get the chance to enter a few dozen bugs, their pain ratings will become far more reliable.

The temptation to assign ‘cost’
You’ll notice that there is no place for ‘cost’ or technical risk of fixing a bug anywhere in the pain score. This is one factor that most developers immediately request. Despite the temptation, I recommend leaving it out.
  • It requires extra effort: 8 times out of 10 the developer actually needs to dig into the code to figure out what is causing the bug before they know how much it will take to fix. If you require ‘cost’ to be figured into the User Pain calculation, you bog down the entire system.
  • 99% of the time it doesn’t matter. The cost of fixing bugs tends to fall on an exponential distribution. Many bugs are one or two line fixes. Others rarely take more than a couple of days. Only a very few are truly killer bugs. Flag the exceptions and use a generic bug velocity to track the rest and your results will be just as predictable as if you had costed each and every bug.
The temptation to automate exceptions
The User Pain system is about automating triage. There is a temptation to attempt to automate everything. What about the 1% of the bugs that have the potential to push your release into the next century? When you do stumble upon a bug that will take more than a week to fix, flag it as a ‘killer’ bug. Killer bugs show up in red on the Pain List and an email is sent to the team.

It is now the responsibility of the team leaders to find a solution. They can design around it, postpone it, or even fix it. Now that they’ve been freed of the burden of triage, they have the time to attack the hard problems with great vigor.

The pain score helps keep killer bugs in perspective so that panic is kept to a minimum. There will be Killer issues that are very low pain. It probably isn’t worth delaying the product to fix these. On the other hand, a Killer issue that has high pain is likely to have a serious impact on schedule and should be addressed immediately.

There is a small lesson here. Never build a system, especially one involving people, that aims to handle every exception. You'll destroy what value the process adds by building in all the edge cases. Instead, allow people to raise an alarm so that smart minds can deal with the exception in a timely fashion.

Conclusion
User Pain remains simple despite all the detail I’ve tossed your way. The team submits and ranks bugs. The system calculates user pain and pumps out a fresh list of prioritized bugs for everyone to see. The team fixes the bugs from the top of the list. Those are the essentials.

Teams thrive under this bug process. There is less thrashing and ambiguity. There is a lot less need for micromanaging every single little bug (and every single developer). With User Pain, the responsibility for creating a quality product is placed clearly in the hands of the team. They triage the bugs. They fix the most important ones early. The process exists to give them all the tools they need to make the right decisions. Again, it is about empowering people, not managing them.

User Pain doesn’t work for every team. Nor does it completely eliminate triaging. Anyone who thinks process is a panacea hasn’t worked in this industry very long. However, with your heart in the right place, User Pain is a substantial improvement over sitting in a room and manually reviewing hundreds of bugs. It makes the team more efficient, helps people make better decisions and focuses the team on building quality into the product.

Take care,
Danc


Appendixes

Bug Maturity
Setting quality bar is great for fixing high pain bugs. However, over time you will build up hundreds of older low pain bugs. Since you are triaging less often, the quality of such bugs can be quite low.

You can alleviate this issue is to add a Bug Maturity factor to the pain score. For every day that passes once a bug is entered, you increment its user pain by .2 points. Over time, old bugs slowly rise to the top of the pain list. You can adjust the rate of bug maturation to match the needs of your particular project.

This has two effects
  • Old bugs are slowly removed from the system: Either you decide to never fix them or you fix them.
  • Small bugs are fixed slowly: Instead of indefinitely leaving small bugs in your product, you end up fixing them in order to meet your quality bars. This helps prevent the accumulation of code cruft.
Tracking Charts


A basic line chart that tracks the number of bugs above your various quality bars works well for tracking bugs. You can compare this to your total bug count as a reference. The ideal trend is that your high priority bugs drop quickly. Warning signs include the following:
  • Focusing too much on feature work: The bug count across the board keeps rising.
  • Poorly directed bug fixing: The overall bug count is dropping, but the high pain bugs stays relatively constant.
Other metrics of interest include:
  • Total Pain: This represents the accumulated impact on the user of all the bugs in the system. Some teams use this as an additional gate to determine if the product should be released or not. It is another way of ensuring that the team doesn't ship with hundreds of small issues that end up causing the user substantial grief. This value is far more meaningful than bug count, but serves a similar purpose.
  • Average Pain: You can get a sense of the general instability of your product simply by looking at the Average Pain across all bugs. A high average pain, especially one near your quality bar, means that you have a lot of work left to do.



Labels: , ,


Read more!

Sunday, May 11, 2008

The Scary List


In every project, there are issues that that frighten the bejesus out of the team. They are so frightening that no one wants to talk about them publicly. The schedule might be impossible. There might be the lurking suspicion that Management does not believe in the project. More commonly, there is a major technical flaw that no one is handling.

Such problems linger over the team. A handful of people hold hushed conversations in hallways or behind closed doors. Secrecy reigns due to fear. There's fear of upsetting team morale. There's fear of losing face with management. There's fear of forcing the project to be killed.

Fear is contagious
Team members are not stupid. A good development team works hand in hand with one another 8 hours a day. People observe body language. They take note when conversation stops when someone walks in the room. They become suspicious and wary.

Fear becomes contagious. Lack of information breeds rumor mongering as others step up to fill in the blanks with conjecture, often wild, about imagined consequences. Blame is bandied about as frightened people attempt to find comforting answers to imaginary scenarios. All the while, the problems do not go away. Instead, they fester, turning healthy teams into a paranoid self destructive wrecks.

It doesn't have to be this way. Nine times out of ten, the scary issues that cause so much panic are completely solvable. They simply need the harsh bright light of public acknowledgment shone upon them.

The Scary List
Here's a technique that I've used on various agile teams to good effect. Every day, we have our team meeting and on one of the walls is a white board containing the heading 'Scary List'. When someone catches whiff of a problem or rumor that could potentially sink the projects, we jot it on the list.

Sometimes, this is enough. Just publicly acknowledging the issue and giving it a name can clear the air. It becomes okay to talk about it out loud. By naming your problem, it is no longer an amorphous mysterious rumor. Instead the team is faced with a defined problem. The ideas start flowing between team members. People discover the sort of solutions that only appear when everyone puts their heads together. Many times, individuals will quietly grok the issue and adjust course to correct it.

Other times, the problem is pressing and needs to be driven to a conclusion sooner, rather than later. At this point, someone volunteers to drive a solution for it. Every day, at the standup, the champion lets the team know how progress on the problem is coming along. In short order, most problems stop being quite so frightening.

Some problems are extra scary because the team feels like they are out of their control. For example, the team might get axed if some other group fails to deliver a particular component. Have a five minute discussion about what is under your control for a particular issue and what isn't. Again just making the assumptions public does wonders.

Next focus on those issues that you can control. Are there contingencies? Can the team flow like water around immovable obstacles? It's that crazy empowerment thang and it works.

When an issue is no longer scary, remove it from the list. It doesn't need to be complete. It doesn't need to be fully planned out. However, if the team no longer fears the problem, then it should be wiped away and converted into more mundane backlog items or tasks on the board. Otherwise, the list loses its impact. No one likes unnecessary fear mongering, even in the form of a process meant to manage fear.

The scary list is:
  • Simple: It is a hand written list. It takes five minutes to create and manage.
  • Public: The list is displayed in a large format in a public area frequented by the team. You can't miss it.
  • Fresh: Only currently scary items are on the list.
  • Actionable: You are either turning the scary items into less scary problems or you are removing items from the list.
Courage
In the very first Agile book I read, there was a large section on 'Courage'. I must admit that I didn't really understand why it was there. The word 'Courage' seemed like such a fluffy bit of New Age nonsense that didn't belong in the crisp world of software engineering.

The Scary List is a simple technique, but it only works on a team where people have the courage to talk about their fears. You need the right cultural environment for this to occur.
  • No retribution against the person who brings up a sensitive issue.
  • The problem, once raised, becomes the team's problem, not an individuals.
  • The exercise is about solving the problem, not placing blame.
This doesn't happen naturally for most teams. They need to practice. You need to lead by example. If blame starts being thrown about, redirect the conversation to the problem at hand. If people claim that they can't do anything, identify the pieces that they can change. Eventually, people will experience success by using the Scary List. With that success comes the confidence to face the problems that frighten them.

Courage, you see, is also contagious.

take care
Danc.

Labels: ,


Read more!

Sunday, October 14, 2007

Lessons about failure


I happened across a wonderful nugget of design philosophy, while reading an interview with Clinton Keith, the head of High Moon Studios' technical department. It bundles up lessons from Miyamoto, the joy of failing fast and the benefits of using Stage Gate-type processes in one delightfully juicy quote.

"If you want someone to fail, you want them to fail fast, before they spend a lot of money. That's how Nintendo was. When I was working on the Dream Team [at Angel Studios], they wanted us to do this DNA-based driving game called Buggy Boogie. You had these vehicles that would eat other vehicles and adopt their powers and morph. It was really cool. But they would sign three month contracts, and Miyamoto himself would say that he did not want any documents. He would just say, "Find the fun, and I'll be back in three months to take a look at what you have."

We went through about three iterations of that. We busted our hump trying different things, but at the end of it, he kept coming back and saying that it wasn't there, and it wasn't fun. We were a new company that didn't know how to make games. After about six or nine months, he came back and said, "You guys have really worked hard, and we see the progress, but we're not seeing the product. But another opportunity has come up for a fantasy golf game, so why don't you guys work on that? In three months, we'll be back. Show us a golf game."

So rather than getting pissed off at us and canceling the contract after two years and millions of dollars, they spent just a tiny fraction of that with a small team and said, "Well, it was just a bad idea." It maintained the relationship with them, so we could go off and do something else.
The Lessons
Here are the tidbits I squeezed out:
  • Give yourself a short period of time to 'find the fun' in a design. Give a small team for a few months to iterate on a new design idea. Your goal is come up with the enjoyable core game mechanics. Toss the lengthy design docs. That can come later. If you don't have the fun core of your game, all the design docs in the world won't help your title.
  • If the fun isn't there, move on. Many ideas are bad ideas. You didn't know until you tried. Luckily game designs are a dime a dozen, so perhaps another one will be more fruitful.
  • If you do fail, it isn't the end of the world. Failure is a reasonable and obvious part of the process of creating games.
Much of how creative people see the world is marred by the success bias. We are constantly surrounded by successful, beautiful creations. It is natural to assume that somewhere out there are people who can just create an amazing game at the snap of their fingers. We look at our lump of an attempt and the comparison can be soul shattering.

What we don't see are the failures, those thousands of experiments that never made the final cut. There is a thread over at TIGsource where they are posting incomplete projects. This is the reality, the secret underbelly of both the marvelous IGD competition entries and many commercial successes. You will fail many times before you creating something amazing.

The multitude of playtests that arise from the plethora of exploratory project will inevitably give you more failures than successes. The smart folks use failures to learn and improve. Failing quickly and cheaply means you'll get to really good ideas faster. The path to success is intentionally strewn with failure. Embracing failure is a fundamental lesson of good design and one that is not taught nearly enough.

So when you look at your feeble, twitching prototype and compare it to the latest vibrant screens of some Miyamoto masterpiece, don't give up hope. He likely went down that path as well. Pick yourself up. Is there are spark of fun in your idea? Can your coax it into a bright flame? If not, you should have no regrets. For there is always the next glorious idea waiting to be explored.

take care
Danc

Labels: , ,


Read more!

Sunday, February 18, 2007

Rockets, Cars and Gardens: Visualizing waterfall, agile and stage gate




The further I dig into new product development practices the more I crave a simple way of helping folks new to the concepts visualize them quickly. In that spirit, I've assembled a little pictorial journey through the intriguing landscapes of waterfall, agile, portfolio management and stage gate. For fun, there is also a description of how you can apply portfolio management techniques to individual agile project as a technique for additionally reducing the design risk.

The concepts illustrated here are readily applicable to most software projects including games, websites and of course full desktop applications.


Software development as a learning exercise
All the diagrams start with two premises:
  • Knowledge is critical to success: A large number of software projects fail because we build software that has little real world value. If you dig into the reasons, even teams with great production skills claim they focused on the wrong issue or they really didn’t understand what the customer actually wanted. When you build the wrong software, you end up with useless code, loss of political support and eternal development cycles. If only the team had reliable knowledge of what to build, they could deliver value early.
  • Most product teams start with very little practical knowledge: Knowledge is great, but we start projects with hardly any. We don’t know exactly what our customers want. We don’t know how our design will work when it runs smack into the complexities of the real world. We don’t know what new opportunities and constraints will emerge in the future. Even when we are experts from a related domain, the best we typically have are theorems, guesses and opinions.
Product development is about rapidly producing targeted solutions that generate maximum customer value. In order to do this efficiently, our teams need to be learning teams that are constantly gathering concrete domain knowledge about customer needs. A team that learns the quirks of its customers, code, and business rapidly will often out perform teams operating without this knowledge. With this concept as a filter, let’s take a look at some typical development scenarios and how they are affect by team learning.

Typical product development of a single product
In most traditional waterfall models, teams gather requirements, develop the product and then test it in order to see if they implemented the spec correctly. Only after they release do they gain any insight into what the customer actually desired. The metaphor that is often used is that of launching rocket towards a destination. You get to aim it once and then you fire it off and pray that you hit the target. All well and good, except A) this is a rocket that burns money for fuel and B) when it misses its target, entire teams (and sometimes entire companies) get the ax.


Now, not all is lost. If your company has enough money to burn, they can try again. There is a good chance the team learned quite a bit about what their customers actually desired by failing in the marketplace, so the next moonshot has a better chance of landing closer to some actual customer needs.



The old joke is “If you want to build a good piece of software, rewrite it four or five times.” Eventually, if you keep trying long enough, you’ll hit a target market that wants your product. Some very big companies have made a staggering amount of money following this dogged pursuit of success. They fail again and again and again, but their few successes fund the further pursuit of new growth businesses. The learning is quite expensive but very valuable.



Room for improvement
This pattern of eventual success through multiple failures has been well established in all sorts of new industries. A few smart companies have identified two basic strategies for speeding up the time it takes to identify a new successful market.
  • Standard portfolio model: Fail in parallel. By trying lots of different options all at once, a few will succeed.
  • Iterative model: Fail sooner. Try something simple, test it to see it works and repeat.
Standard Portfolio Model
If the waterfall process is like firing a rocket at a target, the portfolio model is like firing a swarm of rockets and hoping one hits. The company greenlights a large number of projects, funds them fully and hopes that one of them will blossom into a success. In return for spending more money, you rarely have to wait until version 3.0 to observe success.



This model tends to be highly useful when the target market is poorly understood or ephemeral in nature. For example, in the music industry, when user tastes change every month, it doesn’t make sense to spend three years attempting to evolve an individual album to fit the market. It also tends to be useful when the cost of development is low and potential payoff is high. The incremental cost of green lighting one more project can be easily offset by the incredible amount of money a successful record produces.

The big benefit of the portfolio model is that it allows you to try completely different options simultaneously. Companies are often put in situations where more than one idea has potential and they lack the information to choose which ones to fund. If they were forced to choose only one, they would often miss out on creating multiple product lines. Instead of putting all your eggs in one basket, a portfolio model spreads the risk and increased the chance of multiple innovative breakthroughs.

The Achilles heel of the traditional portfolio model is cost. Smaller companies have difficulty funding one project, never mind ten or twenty.

The iterative model
Smaller teams have learned to maximize their learning opportunities by building lots of opportunity for rapid feedback into their process. The agile software development movement is the poster child here, but many of the same lessons are found throughout lean manufacturing, kaizen and other successful practices used by product development companies across a broad spectrum of industries.



The common metaphor is that of driving a car as opposed to launching a rocket. At every possible opportunity, you are looking ahead and adjusting the team’s trajectory in order to steer towards. Each change may seem subtle, but due to all the rapid cumulative adjustments, the team hone in on their targets quite efficiently.



It ends up being okay that the team makes little mistakes. If they veer off a little bit to the left, that's fine. They rapidly learn it was a bad idea and correct their efforts. The short feedback cycles ensures the big mistakes happen far less often.



Instead of taking 12 to 18 months to create and evaluate a new concept, they build and put new version in front of users every 2 to 4 weeks. They also work in high bandwidth environments where all the team members are close together and close to the customer. Team members converge on and build concensus around good ideas over a period of hours, not months. Teams become experts through intense hands-on problem solving and testing. This ends up building products much more likely to serve real needs than those imagined in Platonic specs by ivory tower experts.

Agile development is favored by small start up teams because the techniques greatly reduce the risk of an individual team failing. If you only have room for one shot at the target, you might as well steer your way to success using lots of rich information instead of launching blindly into the unknown. Long term, agile processes delivering more value sooner, with lower overall risk.

An agile project is intensely focused. In the rush of completing only high priority features, many alternative concepts never get the chance to be explored. Customers, a rather vague and argumentative bunch at best, are required to speak with one voice in the name of reducing noise and thrash for the team. For many teams struggling just to get software out the door, these traits are a godsend. The downside is that there is little concept of strategic portfolio management.



Agile projects can be a bit like a hill climbing algorithm. They will steer towards the closest customer success story, but may ignore a bigger opportunity further away.

Stage Gate: Cross breeding portolio models and iterative development
We can do better than either pure agile or pure portfolio development. The stage gate process borrows a little bit from both iterative and portfolio techniques. You are still launching multiple products, but then you use iterative development techniques to guide each project towards success. You kill projects that fail to meet your goals.



Imagine that we are gardeners. We seed a diverse portfolio of projects, some high risk, so low. As we add team members and customer feedback, the projects begin to flourish, winding their way up towards the sunlight of launch. The core team development practices can be agile since we want to encourage short cycles of rapid feedback. The best ones start to blossom and ripen with obvious customer value. However, some projects are sickly and seem unlikely to produce much of anything.

As gardeners, it is also our job to groom the garden. The weak projects are trimmed back and turned into mulch that helps nourish the most successful projects as well as create a bed for seeding even more promising ideas. The best projects are harvested and sent off to customers, rich with ripe features, bursting with value. We are constantly seeding new projects to keep our portfolio balanced.

We’ve added three items to the traditional agile model.
  • Diverse seed teams: The first is the concept of multiple teams heading in different directions with different goals. There isn’t a single customer, but multiple customers driving multiple potential success strategies.
  • Kill Gates: The second item is the concept of gates every few iterations that kill poorly performing projects. It is better to return those resources back to existing projects than to continue investing. You can think of the success criteria that drives the gates as a set of reasonable constraints. The team is allowed to do whatever it desires as long as it meets the outlined constraints that are measured at each gate.
  • Concept banks: The third is the concept bank where old ideas are stored for possible recycling. You never know when the remnants of an old idea will nourish a strong new project.




You may start out with dozens of projects, but in the end you’ll only launch a few into the market. The combination of trying lots of directions with very small low cost agile teams dramatically increases your opportunities to learn compared to either pure iterative or pure portfolio techniques.

You spend dramatically less than you would in a pure portfolio model since you don’t have the heavy cost of finishing and deploying the bulk of unsuccessful market explorations. All that waste can go back into either exploring new options or building out projects that show success.

You have a higher success rate than a purely iterative model since you are exploring multiple hills instead of climbing a single hill and hoping that it is the highest one around. This recognizes that not all successes are created equal and it can be easy to get stuck in rut serving a niche. By keeping your options open, you can always shift resources around if a truly great opportunity emerges.

One downside is that cost is a little higher than a pure iterative model. You need to devote a portion of your overall efforts to generating and exploring new opportunities. This is typically more than offset by the earlier access to successful products. Another downside is that some long term projects (such as say…a fusion reactor) are culled before they show their promise. This can be handled through careful tailoring of gates so that break through innovations are properly weighted.

The Harvest Process: Applying a stage-gate model to a single product
A single product can benefit greatly from the ideas in the stage gate model. A single product is composed of a variety of separate features, scenarios and audiences. We can treat each chunk of customer value as a separate project and bundle of high value projects as a product. The basic model is the same as the stage gate process outlined above. However instead of spinning off multiple products, your individual teams tend to be focused on critical workflows or user scenarios for a single product.



Returning to the garden metaphor, as you nurture these areas of value, you should always have projects that are ripe for release. Publishing can be as simple as creating a public release that turns off all the experimental projects, leaving only your best features to show the public. I think of this as harvesting a ripe crop for your favorite customers.

As time passes, the little ideas that you seeded earlier will mature and produce a new crop of high value features. The process just keeps repeating, season after season, release after release.

The cliché in our business is that ‘ideas are cheap’. The shame of the matter is that some of the best ideas are left to rot because they don’t come from the right people and the production schedule has no room for creative thinking. The Harvest process builds innovation in at the lowest level possible. Everyone is encourage to come up with ideas and there is a clear and official process by which those ideas can become reality.

Making it work
In order to pull this off you need some basics in place. There is a substantial expense to running multiple projects and constantly integrating their efforts. With the right people, the right process and the right values in place, it is a very achievable goal.
  • Multiple, small agile teams: A single monolithic team has difficulty guiding a multitude of projects. Smaller teams can champion their area more easily.
  • Highly refactored and regression tested modular code: You need to be able to change one element without breaking the rest of the projects.
  • Innovation at the edges: Teams need to be able to innovate in interesting directions without constantly referring back to the ‘master planning committee’
  • Kill gates: If you don’t understand success and kill projects that don’t meet the bar, you’ll end up with a cancerous mass of bizarre, unhealthy features that have grown in random directions. Quality kill gates that actually kill projects are critical to success.
  • Shared design: You still need to present a unified face to the customer. Each team is under additional constraints to share visual styling and minimize conceptual complexity for the product as a whole. These constraints are built into the kill gates.
  • Customer feedback: At every stage, you need to be testing your products with real customers. You should be able to quickly deploy a working product to your customers every day. None of these ideas can succeed without people using working versions of your software and the team being wiling and able to learn from the customers’ reactions.

Conclusion
In the end, any of the techniques outlined above have a chance of working. You need a driven group of people with the right circumstances in place. With the right amount of random luck and persistence, even a waterfall project can change the world. For some fat cats working in bloated large companies, this is enough. However, if failure has a personal cost, you’ll want to look into the alternative techniques such as agile, stage gate, or harvest. Over the long term you’ll notice less overall portfolio risk and your customers will start enjoying your superior, well informed product designs.

If you take away one thought today, it should be that learning is fundamental to software development. We do not live in a Newtonian world where huge mechanical brains can predict the future. Instead, we operate in a constantly shifting organic universe of cultural trends, customer urges, fluctuating cash flow and skeptical sponsors. In order to deliver great products on time, we need to learn from our environment and build a response rapidly and precisely. Portfolio technique increase learning by trying more options. Agile techniques increase learning by shortening the feedback cycle. Build both of these key concepts into your teams and you’ll find that they make smarter decisions and deliver more value to your customers.

Take care
Danc.


References and resources

Original pretty pictures
I was inspired by these image of agile development when I began making diagrams.
http://www.gamesfromwithin.com/articles/0608/000110.html

Small initial teams? Try 20% of one person.
With Harvest and Stage Gate, initial teams can be very small. For example, you could devote 20% of one person’s time for the initial concept stage. That doesn’t sound like much, but if you have every single person in the company devoting 20% of their time to new ideas, you’ll be seeding the equivalent of hundreds of new projects every year for no additional headcount. Google is the golden child of this concept right now, but 3M has been doing it successfully for decades.

Put those hundreds of ideas through a simple checklist that looks for potential customer value. Spin off the best ideas into tiny teams of a couple people and give them an iteration or two to come up with something cool (and testable.) Repeat the process several times and you rapidly gain some surprisingly innovative products .

Iterative techniques
Agile Overview: http://www.agilealliance.org/
Scrum: http://www.controlchaos.com/about/
Kaizen: http://en.wikipedia.org/wiki/Kaizen

Stage gate
Winning at New Products: Accelerating the process from idea to launch, 3rd edition Robert G. Cooper, copyright 2001

Project Horseshoe Report on building games that sell
Many of the ideas in this essay were heavily influenced by discussion inside the Play Dough group at Project Horseshoe 2006.
http://www.projecthorseshoe.com/ph06/ph06r5.htm

Labels: , , , , ,


Read more!

Sunday, November 26, 2006

The Passion of the Developers


A thousand little decisions go into the creation of a great product. Most are not in the specs or master plans. Instead, they are implemented by testers and programmers. Yet, these are the same people that are traditionally lampooned as geeky and out of touch with the target customers.

What would happen if the developers possessed a deep understanding of their customers needs and desires? Suddenly, those thousand little decisions aren’t introducing random noise into the product. Instead, they are pushing the product forward. A single decision might not make much of a difference, but taken en mass, the momentum generated by enlightened developers can result in a massive reduction in design risk. You’ll develop the right product sooner and with fewer errors.

You can’t unfortunately just say “Dude, I looove my customers” and expect to get results. To harness the power of the passionate developer, you need the following pieces in place:
  • Empowered developers
  • Meaningful customer feedback

Empowered Developers
One philosophy is that we need better specs and those damned monkeys need to implement the specs exactly as designed. Better command and control system and more rigorous process is obviously the trick to success. This tends to generate poor results.

Imagine that you are a highly creative individual whose life is reduced to a series of highly detailed checkboxes that you are told to fill out one after another, day after day. If you deviate, you are punished and ridiculed. The results are gun-shy, non-communicative production workers that rely on obfuscation and deliberate bending of rules in order to express their innate creativity. Many wonder why they got into this dreary business in the first place. Others just give up.

When indentured coders and testers make design decisions, they lack both the experience and the desire to do the right thing for the user. They’ve been trained to take orders from management instead of building up an intimate empathy with their user. In the best of worlds, they implement uninspired versions of the features described in the specs.

In the worst of situation, they labor in secrecy, adding a little bit of their own uninformed creativity back into the product. When confronted with the havoc wreaked by Byzantine architectures and convoluted, unnecessary features, the humanoid production resources are surprisingly defensive. How can you be angry? They were just trying to salvage a small bit of their creative souls.

Such feature creep is not the fault of the individual testers or programmers. It is the fault of the team and management for turning a creative and invigorating process into menial labor where talented creators have no other option but to rebel.

We need a different philosophy based on passionate, educated team members that put the user at the center of their world. Developers gain the power to make their own decisions through their deep understanding of their customers.

Simple beliefs
Empowerment is a word that has been stripped of much of its initial meaning by the hippy liberals of the world, but the concept is rather simple. An empowered person is someone who has the ability to make their own decisions.

Let’s look at some building blocks that need to be in place before you can have empowered developers. You need to believe these are true and the people on the team need to believe that they are true. If these foundational beliefs are weak, it is very likely that you’ll end up falling back on command and control decision making structures in the face of emergencies or pressure.
  • Developers are passionate people: Even technical people burn inside with an inner flame of intense creativity. It can be hidden. It can be misdirected. But you must believe that it exists. The other insight here is that developers are motivated by very human issues. Dream, fears, respect and social hierarchy have a greater impact on productivity than economic incentives or logical rules.
  • Most decision should occur at the edges: Management serves a useful role in maintaining group cohesion. They rarely however, have intimate knowledge of the production issues experienced by individual workers. When developers at the edge with customer empathy are encouraged to solve problems that they happen across, a hundred issues that are invisible to managers start going away. A well functioning team is bursting with these small acts of ‘goodness’. Over the course of a project, the momentum results in an order of magnitude improvement in product quality. The most important outcome of this philosophy is that the people who build a piece of customer work flow hold absolute responsibility for it’s usefulness to the customer. “I’m just doing what I was told to do” is the vilest of attitudes.
  • Encourage learning through failure and feedback: The urge to control production often stems from fear of others making mistakes. “The morons are going to just screw things up!” is a common refrain. If you remove the opportunity for people to make bad decisions, you slide rapidly down the slippery path of command and control. An empowered developer has permission to fail. In fact, rapid, low cost failure is actively encouraged since it is the single best way to learn about how to make the product better.

Customer feedback
Now we have a team of passionate people. They are solving problems at the edges and they have permission to fail. Wonderful! We still, however, are missing the most critical element. We need to build a strong customer feedback mechanisms that encourages the team to create a product that matters.
  • Identify the customer
  • Believe in the customer
  • Get close to the customer
Identify the customer
Identifying your customer is where an amazing number of teams fail. They become focused on a bit of technology and imagine that users will just appear out of the woodwork. The whole process of iterative product design requires an identified customer so that you can know when you are on the right track and when you need to change direction.

The first step is to write down your customer definition. You’ll take into account market data, economics, opportunities that you’ve noticed and a billion practical details. It will feel like a very intellectual pursuit. The next step is to go out and find an actual, breathing human being who represents your customer. Customers are not data points. They are people that you can question, observe and befriend. The customer definition is just a tool that helps you narrow down your search. The real win is when you can get your customer in the room with your developers and they can drink a beer together. PBR FTW.

This isn’t easy. You may not be able to find a single person who represents your typical customer. You may end up with multiple customers that represent different needs. You may find that people on the team had very different ideas of who the customer might be. You’ll likely redefine your customer multiple times once you start talking with your initial customer prospects.

It is okay if it is hard. Don’t give up. Choosing your customer is the single biggest product design decision that you will make over the course of your entire product’s development. My belief is that no project should be green lighted for even minor development unless the team has had extensive conversations with an identified target customer.

Belief in the customer
The user of your product must be worthy of great passion. Remember, we are dealing with passionate developers thrive on living meaningful, creative lives. When the target customer is considered unworthy or boring, the development team will feel little urge to empathize or go the extra mile.

Product development is a service to others. The products we produce make someone’s world a better place. The team must be able to explicitly articulate why it is worth spending years of their precious lives providing the target audience with a new bauble. If you can get everyone on the team to passionately proclaim that they wish to serve the customer, you have built a powerful unifying vision that will carry the team through hard times.

It is worth noting that practically any customer can be worthy of great passion. All customers are people with dreams and passions of their own. They have tricky problems that result in very emotional human costs. An important aspect of connecting the passion of the developers with the passion of the customer is to identify the human, emotional aspects of the customer problem and bring those to the attention of the team.

Get close to the customer
Once you’ve identified your customer, you need to figure out ways of gathering information about them. One of the great Zen mysteries of product design is that in order to successfully serve your customer is that you should never give them what they request. So you can’t just grab a customer of the street and ask him to design your product. An average customer is good at identifying problems, but they are poor at prioritizing, understanding larger patterns and generating elegant solutions. That is your job. Your customer is there to let you know when you get things right and when you get things wrong.

The feedback cycle is pretty straight forward
  • Identify a problem or opportunity
  • Build a valuable solution that fits the available resources constraints.
  • Present the solution to the customer and record their response. Do they realize the value of your solution?
  • Use the data to identify new problems and opportunities. Look at the big picture and start the cycle all over again.
There are a plethora of good techniques available for getting closer to the customer. IDEO has their 51 ideation cards, college design courses teach books worth. They operate at different timescales and give you different types of information. Implement multiple ones and see which ones work the best for your particular customers. Here are some of my favorites:
  • Use your own product: Try to replicate what you see your users doing. Mimic the masters so that you understand their thought process by doing. This builds developer empathy with your customers and is one of quickest feedback cycles around. It also however, can be highly biased.
  • Onsite customers: Make a customer or two part of the development team. You get very rapid feedback and lots of cultural transfer into the team. The downside is that customer objectivity can be polluted by constant contact with the team.
  • Observe customers using your product: Set up usability studies and gather both quantitative and qualitative data. Every team should do this, but it can be expensive and feedback only happens in spurts.
  • Hire psychologists and ethnologists to study your customers: You’ll often gather key insights that would otherwise slip by someone not trained in methodical observation. Making these results actionable often takes additional work.
  • Listen to lead users: Some users are actively solving the same big problems that you are attempting to solve, often years ahead of the market. Find them and learn from their successes and failures.
You are always balancing three different aspects of the customer feedback.
  • Reduce feedback cycle time: The more quickly you can fail, the less effort you can put into each failure and the sooner your can correct your course
  • Improve objectivity of results: It is easy to get biased results. Many of the issues dealing with customer psychology are fuzzy in the best of situations and the developer’s own interpretations often dramatically skew the data.
  • Improve clarity of results: With more data comes more noise. Boiling down the flood of information into actionable, meaningful results that the team can understand is an ongoing activity. Many teams collect customer information, put it in a binder and then never look at it again. If you don’t act upon the customer feedback, it isn’t useful.
There is a substantial cost to all of these activities. You’ll need to change your development practices. Daily builds and regular check-ins help ensure that you always have a product that your can put in front of users. You’ll need to spend money. Bringing in users, ethnologists and usability experts adds to your headcount. This is the cost of doing business.

Barriers
I’ve laid out two pieces of a successful product design team. The first is empowered passionate developers and the second great user feedback. Without the first, user feedback is either ignored or fails to permeate the multitude of small decisions that go into building the product. Without the second, the team spirals off into wasteful directionless exploration.

The single biggest barrier to creating a team of developers that are passionate about their customers is the core cultural beliefs of the people already on your team. Chances are good that they rose to where they are today by mastering a traditional command and control style system. They’ve been personally rewarded throughout the years by checking off the check list or writing checklist for others. They got a big fat raise for staying late and adding those 12 new features from the spec. When the product fails, they are told, “We should have added those 6 more features that got cut because developers were so slow.” The weak ones leave and the ones that are great at silently adding questionable features are promoted and promoted again. They are trained, their egos are invested. After five to 10 years of this, an experienced developer builds up an unshakable, unquestioned belief in command and control style development that only falters when they finally die off.

A thousand little successes do not start with a mandated policy. They begin to gain momentum when the role models within the company start acting upon the beliefs in concrete ways. This is where most process change breaks down. People say one thing, but revert back to habitual behavior when the going gets tough. They still believe, deep down, that the old method of command and control is less risky than trusting a team driven by passion and customer feedback.

Only when people on the team have personally experienced success using the new techniques and they’ve adopted new beliefs as a core of their personal philosophy will momentum sweep through the entire team. Several items need to be in place:
  • The right people: You need a core group of people willing to believe in both developers and customers. These folks are rare since the current system isn’t focused on producing them. If you find one, hire him or her immediately. The culture they bring to the team is worth as much as their skills alone.
  • The right situation: Often people need to be in the middle of a failing project before they will consider that there might be alternatives.
  • Time: Smart, strong people do not give up their culture overnight. After years of steady and obvious success, they’ll start to shift. Start with a like-minded core and grow it slowly.
Very few companies have all these items align. I was chatting with one insanely successful game company that follows many of these methods. They’ve given talks on their process. They’ve partnered with other companies and attempted to teach their process. Each time it fails. Imagine a company that consistently releases highly profitable AAA titles merrily giving you their secret formula. Would you take it?

Yet no one does. It would require companies giving up their most fundamental beliefs about how software is designed and built. They would need to give control to the lowest people in the organization, embrace a constant stream of failure and put customers, not their creative urges, at the center of the design process. That, my friend, is scary. Product success is cool, but a shot a being Top Monkey in the current ineffective system is much more attractive.

Why are we in this business again? Where is our urge to build great and wonderful things that serve the world? It seems a shame to let tradition, habit and hierarchy get in the way.

Conclusion
Lately I’ve been contemplating a nuclear bomb of a statistic. 80% of software features are either never used or used very rarely by customers. Anyone who does software development should be staring at that number in shock. 80% of your sweat and tears has little to no customer value. In most cases, customers would be just as happy if you were to give them 1/5th of the existing product and make that little sliver work better. How much time and resources could you save if you only delivered the right features?

This in turn has gotten me thinking about the common software development practices, the ones that fixate on execution risk instead of design risk. What we build is just as important as the fact that we can crap out a product by Christmas.

Passionate developers that empathize and listen to feedback from customers form the foundation of an efficient team for one central reason: They are very good at reducing the risk of building the wrong product.
  • Team members more readily identify areas of design risk that require iteration with customers.
  • Any big design mistakes are caught early through customer feedback.
  • Small decisions are made more rapidly with fewer errors.
You aren’t going to save 80% of your development time since all this iterating does chew up a serious effort. But you may save 50% or 60% of the budget. More importantly, you may end up releasing a product that is of value to someone that you care deeply about.

Passionately yours,
Danc.

Labels: , , , ,


Read more!

Sunday, October 15, 2006

Persistent myths about Game Design

The straw man concept of game design involves the sole genius game designer who writes his thoughts on a golden tablet and passes it down to the production minions to build.

There are several beliefs about the process of game design and development evident in this stereotype.
  • Heavy upfront design and preproduction are critical to the creation of a great game.
  • When these activities are not done early in the process, there will be mistakes made later in the development that are almost impossible to correct.
  • A single individual must drives the creative design process. Otherwise there will be a lack of vision that cripples the project.
To this day I still come across designers who enthralled by such philosophies. How often have you heard comments like:
  • “If only we had more preproduction time, we wouldn’t have run into this problem.”
  • Or “If only I had been given more creative control, the product would have turned out better.”
These statements sound quite reasonable and many of us can bring up examples where such practices appear to have worked. However, as the industry becomes more experienced, many have come to understand that traditional upfront design presents a myriad of problems including delayed schedules, conservative design choices and years of our lives spent producing game designs that customers simply don’t want.

Our development beliefs have deep roots in the creative processes that have been historically promoted as the right thing to do. Let’s look back at why they are so prevalent and why they fail when applied to modern game design and development.

Historical Roots: Software development as an engineering discipline
Back at the turn of the century, manufacturing believed that heavy upfront planning was the key to efficiency. The mantra was “Do it right the first time”. There is a rich history of ‘time and motion’ process development where learned men with stop watches would plan out manufacturing steps performed by their laborers to the second. Software development, in an attempt to add rigor to their fledgling discipline adopted the manufacturing philosophies whole heartedly.

The following are wisdom gleaned from this ancient era. In small doses, they are quite beneficial. When they are followed dogmatically, time and experience has proven that they destroy projects.

Myth: “Upfront design reduces risk”
We all are familiar with waterfall style preproduction, production, post production cycle. The idea is that by planning upfront you think through problems early and avoid stupid mistakes. In this line of reasoning, the longer you think about a problem, the more edge cases you’ll discover.

When teams are given plenty of time for upfront design, they will typically design elaborate systems for problems that do not really exist. The interaction of software with users is notoriously difficult to predict. Problems emerge from the most unlikely areas and other issues that take of dozens of hours of planning end up not being all that important. When a team plans in isolation from feedback, the result is bloated software that either poorly solves the actual problem or does not connect with the users needs.

A better technique is to test your ideas early and often.
  • Build working systems that you can show to users as early as possible. This allows you to gain real world knowledge at the earliest possible point.
  • Try multiple paths early in development. This results in the cross fertilization of expert knowledge necessary to create truly innovative solutions.
  • Do the simplest thing possible. Instead of worrying about every contingency, do the simplest thing that will solve the problem at hand. If you get feedback that it needs to be improved, go for it. This dramatically reduces feature creep.
  • Defer decisions as long as possible. The longer you can gather information, the more likely you’ll make the right decision. This may smack of being wishy-washy, but it is an attitude that encourages building flexibility into the system and allows you to adjust to feedback more easily.
  • Iterate: By building rapid feedback cycles into your development process, you’ll converge on an optimal solution far faster than you would if you spent time in planning.
  • Include customer feedback. Play that game constantly. Watch others play the game. Collect real world data and share it with everyone on the team. Act on the problems you witness.
There are two forms of risk, execution risk and design risk. Execution risk is the risk that the project will not be completed as planned. Historically, it is what our fledging industry has been overwhelmingly concerned with. “Can we even make a game?” Design risk is the risk that you will complete the wrong product. This is where mature developers should be focusing their effort “Am I making the right game?” Upfront planning focuses on reducing execution risk. In reality, long preproduction delays the production effort and results in a product that is disconnected from the needs of the market.

Rapid iterative development reduces both production risk and design risk. Design risk is reduced by constantly reconnecting with the customers. Production risk is reduced because you find that you often need to do a lot fewer features in order to satisfy the core customer’s needs. It is amazing how you can cut production schedules by simply doing half as many features. Just make them the right customer focused features instead of the flights of fancy of a planner locked in a dark room.

Myth “The cost of change increases exponentially as time goes on”
Many conservative design decisions are motivated by fear of change. We are told that a change late in the process will cost 1000 times as much as a change early in the process. The argument is that it takes five minutes to write a spec, two days to program the feature, two weeks to test it before deployment and a month to write a patch that fixes a problem after deployment.

This belief encourages logical people to optimize their behavior in peculiar ways. The first inclination is to create highly detailed plans of action that are presented to the production teams as the Bible for the game. Some common problems include:
  • Locking into design details before real experience based information is available
  • Locking the development team into a design that is guaranteed to be wrong.
  • Discouraging ‘Aha’ moments that occur later in development as the team learns about new and exciting attributes of your emerging gameplay. If it isn’t in the plan, it doesn’t get developed.
Second, you want to choose the lowest risk designs as possible. Changes are expensive and low risk designs are less likely to change. The result is a glut of surprisingly narrow minded design decisions made early in product development.
  • Choosing well known designs that are already implemented in other projects over new designs that have multiple unknowns.
  • Discouraging experimentation after preproduction in the belief that it will lead to difficult to plan for innovation and feature creep.
It turns out that the act of attempting to control the cost of change through waterfall techniques bloats the cost of even simple activities. The team puts on spec blinders, focuses on low risk designs and fails to consider the surprising solutions that appear when smart people slowly attain mastery of a complex system. The process itself of excess upfront planning causes the exponential cost of change, not the inherent nature of the work.

When you take the same activities in put them in a software development environment that emphasizes rapid feedback cycles and modern coding techniques, you see a much flatter cost of change. This requires systematic changes throughout the team, not just in the area of game design.

Introduce flexibility into programming activities:
  • Refactor religiously. Use a language that lets you refactor and make refactoring part of your daily coding habits.
  • Use modern source control and coding standards.
  • Reduce rigid code ownership. This ensure that the right person is available when the time comes to make a change across multiple areas of the program.
  • Encourage unit test suites. This builds a safety net that allows programmers to experiment without worrying about irreversibly breaking existing code.
Introduce flexibility into artistic activities:
  • Create highly reusable artwork that follows a standardized format. Procedural content is a must. One shot disposable artwork such as cinematics or in-game cut scenes should be avoided.
  • Reuse artwork from previous projects. Prepare to ship with some of it.
  • Use easily replaceable dummy artwork to prevent bottlenecks during rapid design iterations
Introduce flexibility into design activities:
  • Keep design documents simple. Try post-it notes stuck on a wall in a team area.
  • Use prototypes instead of design documents. If you can’t play a game mechanic, you are just waving your hands. This is often worse than useless since it can hide really bad mistakes for surprisingly long periods of time.
  • Rely on reusable game mechanics instead of one shot level design. Think football, not Disney rides.
  • Be very willing to cut scope. 80% of the designs that you suggest should not ship. Reevaluate your design after everything iteration.
There will always be major features that are difficult to change, but these should be the exception, not the rule.

Myth: “A specialized workforce organized into functional silos is the key to productivity”
The heart of the concept of mass production is that complex system can be decomposed into a series of easily manufactured pieces. The production of these pieces can then be streamlined and created by a specialized workforce that sequentially processes each step of production. An underlying assumption is that if each one of those pieces is built in a high quality fashion, the assumption is that the system as a whole will be of high quality.

Many game developers embrace this strong division of labor. The game designers sit in one area of the office and plan the game. The programmers sit on another floor and implement tools and game systems. The artists are outsourced. Each group is managed based on meeting individual goals associated with their specialized task. The work then flows through each team in a regimented fashion with very narrow bandwidth communication between each group. The design documents and Gant charts describe who should pass off what materials when. And the whole system hums along like a giant clockwork machine.

This does not work. Developing each aspect of a game in a vacuum leads to the following:
  • A rigid inability to change. When an artist realizes that they can improve their process by changing up a few steps, there are often few channels of communication or incentives for communicating these changes to other groups. Often each silos will actively attempt to kill change in other silos in an attempt to “stick to the plan.”
  • A lack of an overall vision for the product. When change does occur (and it inevitably will), each silo begins to drift out of sync with one another. If the drift goes far enough, each group will end up with distinctly different visions of what the end product should be.
  • Isolated silos of workers protecting their own turf. With separate goals and visions comes conflict. Dominance games, unhealthy compromises and apathy are common.
  • Massive integration costs. Often games are integrated together at the very end of the production cycle. Many projects that are not playable in whole until only 20% of the schedule is left. The cost of shoehorning divergent systems and teams together after months of primarily independent development tends to bring up substantial technical and personnel challenges that no one predicted.
Having a team filled with skilled experts is a good thing. It ensures that you don’t end with jack of all trades, but master of none. But once you get those experts, you should seat them together and encourage cross training between disciplines. This ensures a shared vision and helps reduce sub-optimization.
  • Sit cross functional teams together. Being within shouting distance of your artist or your programmer increases the bandwidth that you have for communication by a factor of 1000. This means that the vision for the project is constantly reinforced. It also means that experts from each discipline can come together to solve very tricky problems in a relatively straightforward manner. You’ll come to decisions in 1/10th the time and your decisions will be more well rounded and better informed.
  • Make each team no larger than 10 people. When you get more than 10 people, communication suffers. Maintaining group cohesion requires more management and ape-like ‘social grooming’ time. Teams of around 5 to 10 typically have the highest per person productivity. String multiple teams together with a system like Scrum if need be.
  • Build working games, not isolated systems: Integrate early and often. This goes hand in hand with iterating rapidly and prototype driven design. Each cross functional team has everyone they need to make a complete working game from the start of development. By integrating early, you’ll exercise your development pipeline and you’ll encourage each discipline to being working together early in the project.
Myth: “There is one game designer who holds the vision for the project”
The artist of the previous century has been lionized as a lone creator who imagines a visual in their head and single handedly copies it onto paper, film or clay. On the traditional art project, single artiste owns the creative process from concept to execution.

Shakespeare, Picasso, Rodin dominate our art history. As tribal apes, our innate bias is to assign ownership and dominance to a few individuals at the top of the social pyramid. If the vast majority of publicly celebrated creative works of the past have been attributed to a single artist, then surely game design must follow the same model?

The only problem here is that most games are not created by a single person with genius level skills in a mature craft. They are team activities that involve the creation of complex, new-to-the-world interactive systems.
  • Games represent a class of problems that benefit greatly from collaborative problem solving and use of specialized skills. It is difficult (though not impossible) for a single person to make a modern game. Teams typically produce more enjoyable, more fully realized experiences.
  • Communications issues dominate the creative process, not individual creativity. It is more likely that 10 moderately creative people who can communicate will create a great game. A massively creative person who cannot communicate is outside the game design sweet spot.
  • The ability to solve new problems is more important than the mastery of craftsman-like skills. Game design involves exploring and manipulating complex system that have highly unpredictable results. Every day presents a new set of problems that no one in the entire history of the world has ever witnessed. This is very different than a person getting to spend ten years working in a stable medium such as oils or clay that haven’t changed in the last century.
These are big beefy differences between game design and traditional art that have a deep impact on the process of creating a game. When your team insists on following the model of a single grand designer, expect the following:
  • Communication issues come to a boil: Team conflict increases as people feel they are not being heard. Talented team members may leave the team.
  • Artistic quality is low: When one person makes all the decisions, they tend to ignore the expertise that is inherent in all the other team members. A thousand little decision that could have been made with expert knowledge are left up to a single person who cannot possible understand all the nuances. The result is Star Wars Episode One.
So when you get down to the practical matter of making games, the historical metaphor of the lone creator offers few if any practical lessons. Certainly, the tale of the lone creator is appealing and perhaps inspiring, but by no means does it represent the pragmatic sweet spot of modern game development. Instead consider the following:
  • Train everyone on the team in design. There can still be someone who champions design concepts, but everyone on the team should know the techniques, practices and philosophies of game design. If the lead designer is hit by a bus, the rest of the team should be able to quite merrily finish up the project.
  • Use a coaching model to lead the team. Instead of saying that there is a king figure from which all design flows, push out responsibility to the team members and use leadership positions to encourage learning and transfer of wisdom. The best game design ideas will come from the devs in the trenches. It is the leader’s job to unblock the flow of ideas from bottom, not push ideas onto the team from the top.
  • Reward the team, not just the leadership: Everyone on the team makes the game, not just a few leaders. It is easily to fall into the trap of ascribing success to the inspired leadership and failure to the team. Act against the biases inherent in our damned monkey genes to prevent stratification within the team.
What happened in the rest of the world
Much of this advice may seem counter intuitive or impractical. The majority comes from hard won experience trying to fix systems that were broken.

Just as software development was discovering the tenets of “Do it right the first time,” manufacturing was abandoning intensive upfront planning and design. About three decades ago, Japanese firms introduced the concept of lean manufacturing with an emphasis on strong team work, continuous innovation and just in time delivery. Decisions are made as late as possible when the most information is available. The workers at the lowest levels of the organization drive innovation, not planners detached from actual work conditions.

Lean manufacturing brought about a revolution in manufacturing efficiency. Quality and innovation increased while development times plummeted. The massive planning phases and risk avoidance of previous generations turned out to be more of a burden than a benefit. Those who did not adopt the new agile methods of production where driven out of business.

I expect a similar transition to occur in the game industry over the next decade or two. The market abhors waste and our industry is currently very wasteful of its development talent.

Lessons from the straw man
There are many game developers who no longer build their games using massive upfront design with regimented silos of specialized workers. Throughout the industry there are bright spots of individual developers doing great things to make the process of game design more agile and responsive to customer needs.
  • Valve is known for its cross functional Cabal system.
  • Spore is known for its strong prototype driven development and procedural content
  • High Moon is known for its agile software development techniques.
We are all trying to design a great game. It is worthwhile to note what does not work. More preproduction is not the answer. Bigger teams are not the answer. Finding the one right game designer is definitely not the answer. Game developers need it burned into their skulls that project that follow the “Get it right the first time” philosophy tends to be devoid of innovation, over budget and lacking creative cohesion.

The process of game design is undergoing a fundamental shift. As an industry, we moving beyond our primal fears of failing to complete a game and are beginning to ask “How do we build the right game?” In order to do so, we must throw away many of our borrowed practices.

So if game design is not the act of a creative genius passing down tablets to the team from on high, what is it?
  • Game design is about listening to customers and getting rapid feedback on working software, not writing fantastical design novelettes.
  • Game design happens throughout the development process, not just in the pre-production phase.
  • Game design is a collaborative team activity, not a solo activity.
Take care
Danc.

References
Lean software development: An Agile Toolkit for Software Development Managers
This book is a good overview of how lean manufacturing concepts can be applied to software development. Of particular interest is the concept of ‘waste’. Highmoon claims that some of the exercises suggest a 50% reduction in cycle time is possible in their current development techniques. That is awesome.
http://www.amazon.com/gp/explorer/0321150783/2/ref=pd_lpo_ase/002-8289134-0560847?

Valve’s development process
Valve is a fascinating example of applying many of the principles mentioned in this article to a linear “Disney ride” FPS.
http://www.shacknews.com/extras/2006/060806_valve_ep1_2.x
http://www.gamasutra.com/features/19991210/birdwell_01.htm

An illustrated explanation of reality of game design planning
I love these pictures. They are so spot on.
http://www.gamesfromwithin.com/articles/0608/000110.html

Agile development blogs
http://www.gamesfromwithin.com/
http://www.agilegamedevelopment.com/blog.html

Labels: ,


Read more!