Agile Martial Arts to Prevent The Project K.O.

Recently, I have been having a number of discussions with colleagues about our projects and I noticed that there was a similarity in how we were discussing using the communication and feedback tools of Agile Development with concepts of Martial Arts.

When training in martial arts, one comes to learn that one of the most powerful tools one has in dealing with an adversary is to control distance and time. One can either increase distance in order to give one more time to analyze and deal with an adversary's attack or one can reduce distance and/or time to limit your adversary's options and ability to develop an attack.

Similarly on a project, surprises are seldom a good thing. However one cannot know everything which is going to happen. Change management strategies have evolved to try to deal with this, however, traditionally these have been largely reactive... like a novice at Close distance flailing wildly to block each attack. So how do you control time/distance to problem fallout on a project? 

This is done via the use of feedback loops of varying frequency. Feedback loops of lower frequency can give you the time to "step back" and assess changes before rushing in. This can called a Distance/Time giving action since they give you more time to assess a developing problem, but that means they may also allow the problem to develop further. Feedback loops of higher frequency allow you to address issues immediately, preventing them from ballooning into larger problems or even implementing proactive solutions to stop the problem in its tracks. These are Distance/Time takers, since they shorten the time allowed for a problem to develop. The number and roles of individuals who receive and deal with the feedback initially also plays a part in whether a technique is more giving or taking in nature. Generally, lower number of individuals whose roles are closer to the source of the problem will be a taking action, while the converse is more of a giving action.

Some Distance/Time giving techniques (lower frequency loops):

• Daily scrums/standups
• Iterative development/delivery
• Regular demos
• Regular retrospectives
• Pairing

Taking a step back to gain distance and time is one of the simplest defense moves. Appreciation of the different threat levels each distance presents and the ability to back-out to a safe distance is something which is taught to beginners. Similarly, the techniques for gaining Distance/Time in Agile Development are simple enough that any team can begin to use them right away. 

For example, a team is following a more traditional project management approach with few touch points across the team. Perhaps the PM only reports to a board  the status of things as dates for delivery of milestones occur. At this point, it is usually too late and if a problem has occurred, the PM is delivering the surprise bad-news. Even worse, a milestone is delivered only to have the product owners report all the surprise deficiencies. Obviously, both of these are uncomfortable for all involved and panic may often set in. By establishing short, frequent and regular meetings with the team including all stakeholders, a team can get the opportunity to see the problems coming and properly deal with them. They are effectively giving themselves the time to deal with the problems rather than rushing headlong into the fray. 

A recent client complained that historically, every time they tried to do a deployment the tools the development team used seemed to create surprise errors. Moreover, items were added to the new environment without the client's prior knowledge. These would not necessarily break anything, but they did add confusion and wasted time. In order to address this we established regular meetings with tech leads, the deployment lead, the clients and testing. At the meeting, proposed pending changes were discussed. A draft-deploy was reviewed for surprise dependencies which might be pulled in by the deploy tool. Team members who previously never spoke to one another now communicate regularly around a problem they were all experiencing from different perspectives. Out of this, the team has developed a much more rigorous deployment practice, surprises have been eliminated, and all parties have gained a much better understanding of their deployment tools and an appreciation for each others difficulties in deployment. It is not as much a typical Change Management/Approval board as it is an opportunity for the interested parties to communicate. Meetings never last more than half an hour and usually are about fifteen minutes, but we meet at least once a week. While not one of the traditional Agile Team Meetings, it follows the spirit of frequent cross-communication which is so vital to the Agile Approach. 


Some Distance/Time taking techniques (high frequency loops):

• Automated Testing
• Test Driven Development
• Acceptance Criteria/Tests
• Test First Development
• Pairing
• Continuous Integration
• Daily scrums/standups

Distance/Time taking techniques are usually more advanced in Martial arts as they require a bit more training and discipline to do safely. So too in software development. Most of the ones I listed here obviously employ development practices which software engineers will need to hone if they have not done them before. However, once proficiency is obtained in them, these practices can be life savers and even keep at bay the dreaded pager. Utilized well, these operate as a means of clinching problems by stopping them well before they develop into much larger problems in a more devastating environment. Use of Automated testing with a modern CI server can not only identify problems in with new code early on, it can also help make sure that once a bug is fixed, it stays fixed. One of the mantras on a team I was on was "Find a Bug, Write a Test." Basically, whenever a bug was reported, the developer wrote a test for how the solution was supposed to behave. Of course, until the bug was fixed, the test would continue to break. However, the bug would likely never resurface again since our test coverage of our code had now improved. This of course requires the developer discipline of running tests before committing, as well as a Test-First mentality. Without those disciplines the team is back to reactive flailing.

These techniques do not always stop problems from occurring, but like the clinch, they can prevent the problem from developing into a more lethal one. A team with good CI, TDD, and Acceptance testing practices for instance, can deal with and minimize the impact of major architectural changes or even severe problems introduced by changes in third-party software. They can tackle the severe problem in manageable chunks at a level of relative safety. 

I listed daily scrums and pairing under both because depending upon the situation these techniques can act both to give time to deal with potentially developing problems as well as to stop problems from starting in the first place through open communication. The latter is usually the case where someone either states what they are working on or having difficulty with and someone else on the team either gives them a solution or can help them avoid pitfalls they have experienced before. As it happens at such a frequent schedule, the time for problems to develop is often completely eliminated - effectively clinching the problems. Scrums and Pairing as time gaining operate in the sense that by approaching problems as a team tends to prevent us from making rash decisions.

A final note: you will notice that I do not say Pair Programming... while I like Pair Programming, many of my colleagues have noted that pairing is a powerful tool for all team members. I have seen PM's, BA's testers, even Product Owners pair on deliverables with great results.

"Our team is too _____ to be Agile"

Team dynamics is a common challenge with teams new to Agile Development. This often presents itself in comments such as "Our team is too big/small/remote/unfocused/unavailable/divided to use Agile".  Teams which have not thrown in the towel and who are still earnestly struggling with this might ask questions such as:

"What is the 'right size' for an Agile team?"
"Can we use Agile Management with remote teams or remote members"
"Does an Agile team need to be co-located?"
"Do agile team members need to be 100% allocated?"
"How do you manage multiple Agile teams (with the same backlog)?"

With the exception of the last one, these are all assuming the definition of a team as being a group of people representing the various involved skills and interests necessary to produce a solution against a shared backlog.

In essence, these are really all the same problem: how to synchronize the team members effectively. Not surprisingly, the first pain point that often comes up is around the Agile ceremony focused on team synchronization: the daily Stand-up or Scrum.

How does this look in the previously mentioned scenarios:

Small team: "We (I mean 'I') know what we are (ahem... 'I am') working on, so having a stand-up feels like a waste of time."
Sometimes a team can be small enough that it really does keep itself in synch almost naturally. So maybe you don't need to have a formal daily synchronization. This is however where the other important output of a daily stand-up might need special attention: updating information radiators. Just because the core team are all in alignment, make sure that you keep any other stake holders in the loop so that they can help or give guidance/direction where needed. Some sort of regular check point is also good to make sure that you have everyone's attention in case an impediment comes up that needs to be dealt with higher up.
Advanced Tip: try synchronizing not just on tasks, but on your common goals. A short daily re-dedication to team goals or values (perhaps those noted in a retrospective) can be very powerful even for small teams which are otherwise synchronized on the work being done.


Big Team: "We have so many people involved that our stand-ups take an hour and we never really feel that we have a common direction on the work. Also, because the team is so big not everyone shows up all the time"
First, ask yourself whether all these people are really needed for the work being done in the given Sprint. If not, can their needs/interests be met by less frequent special meetings to synchronize around their particular interests? Some teams get bogged down with extra BA's, Testers and even Scrum Masters and Developers who are really there to collect information for a tangential team. While communication and integration across teams is important, they are really more like additional Customers to your team and as such, should not be involved directly with the core teams synchronization. Once synching is completed, information they need can be more appropriately communicated in a separate meeting.
If everyone at your Stand-up is really part of the Sprint work, can the project backlog be carved up and the group split into two teams? This may take a little planning and grooming as well as coordination against the program/product backlog or release plan.
Agile can be done with large numbers of people. Team size however often tends to become less efficient beyond 6-8 developers plus the rest of the team. Usually this means 1-2 of each of the following: BA, QA/Tester, Scrum Master/Team Lead. The trick is to carve up the work and coordinate the effort such that value is still being regularly delivered. This is often done around value 'themes'. If team size starts to interfere with delivery of your commitment every sprint, think about splitting up the team.
Advanced Tip: Splitting up the backlog and team is simply an organizational abstraction. Given you have decided to split the backlog into two teams for efficiency, if you hit a sprint where it makes more sense to work as one team to deliver value go ahead and recombine for that Sprint.


Team with remote/non-co-located members: "It is hard to find a time to meet. Communicating with remotes loses much of the context so stand-ups become more of a status report. Not everyone shows up regularly. Not really sure what team members are working on. We feel like we are constantly fighting for synchronization, not just at stand-up."
I include non-co-located members in this category because one can feel like a remote if one is across the building, the street, or the city. The problems can be the same. When people work in close proximity to each other, there is a lot of high-bandwidth communication which goes on around discussions about tasks being done. Much of this can be lost by team members who are too far away to hear these conversations. This dis-synchronization seems to dilate with the difference in time zones. A team with members in the same time-zone +/-1 hour may be able to make up via various communication tools such as IM, Skype, Google Hangouts, Google Docs, and remote pairing techniques. Beyond that, the difference in time zones begins to add overhead to delivery. The benefit of the remote members has to be weighed against the cost to productivity on the Project as whole. One way to counter this friction is to gather remotes into teams which are closer in time zone (ideally co-located) to each other than the main team and carve off some of the backlog just for them, similar to dealing with a too Big Team. Similar also to the Small Team, regular and frequent updating of information radiators and publishing them so that remote members can see them is crucial. If it can be partially automated (such as build status, or even Sprint burn downs/ups) all the better. This is the type of team where having a digital Story/task board or at least duplicating the Story/task board digitally may be highly beneficial. Advanced Tips:  1) If you can afford to bring the remotes "to the mothership" at least for a sprint or two in the beginning, that can help solidify the team. Some teams bring the remotes "home" at regular intervals. 2) Remote pairing is a viable technique. If you have high bandwidth internet, you can pair using tmux or google docs (depending on what you are working on) while speaking over skype or IM. Particularly for paired programming, however, you should make sure that the remotes are good at pairing in person, so they know what to adjust to when pairing remotely. An excellent presentation on pairing done right can be watched here. Joe Moore also has lots of good tips on his remote pairing blog.


Team with partially allocated members: Similar to the Team with remote members with the added "Partially allocated members are going to A LOT of daily meetings."
Sure, it is nicer to have 100% allocated teams, however, team members do not need to be 100% allocated to an Agile project for it to work. They do need to be 100% dedicated to working on the Agile project for the percent of time they are allocated. If a team has a member who is allocated 50% to a project, for 20 hours of their 40 hr week, they should be working solely on helping the team meet its Sprint commitment. This is one of the reasons I do not like assigning tasks during planning but rather prefer to just place the stories in priority order and allow the team members to pick up tasks as they finish other ones. In the partial allocation scenario this avoids someone who finishes their task early, realizing their other tasks are blocked, and then just spending their time trying to get ahead on their other project. What should happen is they should be empowered to pick up something else in the Sprint backlog to help the team deliver the value it has promised.
Another guideline I have found is that allocation beyond two projects usually lowers productivity. The scheduling of time to synchronize with the team can actually become harder with too many allocations than it is with remote members. If they are allocated across 3+ other Agile projects, think about how much time they are in meetings rather than working for any of the projects' deliverables. Furthermore, the context switching demanded on them is a huge drag on their ability to engage meaningfully.
Due to the caveats and the hidden losses of productivity, this is why many teams just decide to have 100% allocation and avoid the potentially costly headaches.
Advanced Tip: If you have a program with many projects who share many of the same resources, consider consolidating those shared resources and carving the backlog such that they are only spread across one or two projects. With two projects, someone can work in the morning on one project, go to lunch and finish the day on the other project. This is much more manageable for all involved.


Multiple teams (or remote teams) with the same backlog: Similar to the Big Team with the added "We often feel we are stepping on each others toes. We find it hard to break up/hand-off/work on stories and deliver meaningful value." Add the complexities of the Team with remote members if one or more of the teams is remote.
The difficulty here seems to come from the fact that being too closely joined around a backlog, the teams are not independent enough to self-organize and drive value. Try to carve up the backlog into functional chunks. This may be almost a sub-backlog. If the teams are working this way, then also ask whether synchronization with both teams fully present is of real value or whether each team should synchronize internally and simply report out at a separate meeting regarding integration points. This can keep the team focused on the work it has in front of it, instead of the work the other team is doing which it really cannot impact anyhow. Having good testing between these teams can provide an excellent interface or bridge.  This can allow teams to develop independent of the deliverable of the other team. When the other team completes its portion the testing framework is already there to verify the final integration. You may want to up your compliment of testers to at least one per team with possibly an additional one to facilitate cross team testing. Updated information radiators are also key to helping each team stay on top of what the other is doing without being over intrusive.
Advanced Tip: One of the dangers of carving up a backlog and using multiple teams is a siloing effect. To combat this rotation of team focus can be a powerful tool. If teams are all local, they can implement this via rotation of members. This provides a gradual change that has less impact on each teams velocity. If one or more teams are remote, this can also be done, but you will likely wish to switch the whole teams focus. In this case, you can mitigate the drag of context change by setting up a temporary exchange program. Let us say Team A is working on widgets and Team B is working on sprockets. When they switch a member of Team A continues to work as a remote pair for Team B for a Sprint or two. In this way Team B can get some of the expertise of Team A's prior work on widgets. Similarly someone from Team B would remote pair with Team A for a sprint or two sharing knowledge about sprockets. Again, well written tests also can greatly facilitate this transfer of knowledge and responsibility.

9th century incremental development

Appreciation of the incremental, pragmatic approach used in Agile Development from the annals of history:

"The just man builds on a modest foundation and gradually proceeds to greater things." - unknown circa 9th  century.

This saying is used by Asser to describe the philosophy of learning of King Alfred the Great.

* 'just man' may also be more modernly translated as 'man with sound judgement'.

Homer on Pairing

(From the Iliad, Diomed in giving his counsel on how the Achaeans might capture some intelligence on the Trojan plans):

"When two men are together, one of them may see some opportunity which the other has not caught sight of; if a man is alone he is less full of resource, and his wit is weaker."

Prezi on Writing Communicative Tests using BDD, Cucumber and Rails

Writing Communicative Tests with BDD, Cucumber & Rails on Prezi

Eat your meat AND your vegetables!

No matter what dietary fads are upon us, most of us realize that we need to eat a balanced diet.*

Why is it in the software industry that I so often see teams binging on just one thing when it comes to testing?

What I mean specifically, is testing frameworks. In the Rails world the frameworks du jour are RSpec, Cucumber and Steak - with a little Capybara sauce. Mmmm sounds tasty!

RSpec has become a solid core for all layers of testing with the other parts. Cucumber was initially created to make the BDD'ness more domain specific... so that customers/project owners/BA's could read and even write them. Unfortunately, many Gherkin examples in the wild tend to be anything but non-coder friendly. This seems to have occurred due to many factors which may come to play on a team. 

One of these factors seems to be part of what has attracted many people to Ruby on Rails in the first place. You often hear Rails coders talking about how they love it that RoR 'gets out of the way' and 'just lets them code'. This is definitely laudable when what is getting in the way is technical scaffolding or other technical encumbrances. It is like using a power tool over the old manual ones. Where it becomes a problem is when what is being 'gotten out of the way' is meaningful communication and collaboration across the team. Teams who have tried to get their 'business folks' involved through BDD have run into the problem that either the business loses interest or can't write the tests well enough. I believe at least a good portion of this is due to how the tests are being written. They are often too low level - basically a series of instructions for filling out web forms, clicking buttons, etc.  The folks behind Cucumber have tried to force a solution by removing one of the culprits they provided early on: web_steps.rb. Web_steps was a sample collection of step definitions around many boring day to day activities. Instead of serving simply as an example, teams baked their tests around it - creating many awful to read, low level test suites.

While I applaud the removal of web_steps on the part of Cucumber, it is having the oposite effect. Teams are just moving away from Cucumber to Steak (because it is more 'coder' friendly) for their Acceptance/Integration tests. What lies at the core is a one-two punch of badness that I am seeing more and more: Teams are confusing Integration Testing with Acceptance testing AND coders are writing those tests.

Acceptance tests should be written in the domain language of the business. They are meant to be high level - to provide the coders a place to start. 

Integration tests should be written to test across 'the stack' and as such are more focused on the implementation details. These are therefore in the more code like language of the implementation domain.

Unit Tests test the code directly... everyone knows the coders are supposed to write these. 

Cucumber is good for BDD acceptance tests. Steak is great for integration tests. Eat your meat and vegetables!

Let me suggest a possible test driven development team and their roles in testing under the above approach.

"The Business" (Story owners/Customers/BAs/etc) - write high level Acceptance Test Scenarios - These could even be part of the story. In essence they write the Cucumber Tests. (Someone else might actually cut and paste the tests into a 'feature' file, but you get the idea).

"Coders" (Software Engineers/Programmers/etc) - write the step definitions behind the Gherkin. Step definitions can be more implementation specific. Writing these can be the first time the coders actually start doing Test Driven Design. Then the coders write their Unit tests however they see fit.

"Testing Team" (Test Engineers/QAs) - write Integration tests using Steak, etc.

The above scenarios are based upon testing using Cucumber and Steak. It could just as well apply to the myriad other testing frameworks (Fitnesse, JBehave, Selenium, etc). The essence is that integration and acceptance testing should be separate endeavors.

I can hear the objections already.  

"We don't have a testing team.":  Fine, there are projects which are small enough, simple enough where the coders serve this role. The point is, integration testing is a different animal from acceptance testing.

"Our business people are not involved/would never write tests)": Really? Who writes your stories? Maybe they were driven away by previous bad cukes. Listen to your business people describe the functionality they want. "So I am on the home page and when I go to create a new account then I should get some indication that the account creation succeeded". That is practically Gherkin already! If they won't physically write the tests, then the BA's could certainly be following them around 'taking dictation' as the acceptance tests drip from the Owners' lips. 

On the other hand, if your business people are not involved, it is likely you have a deeper issue than what testing framework you are using. Do you have an Agility Coach?

"That is totally going to slow us down!": Again, fine! Speed by itself is not a good thing. Without direction it simply means you are going to drive off the cliff faster. If you are a business owner and the 'acceptance tests' you are being fed make you want to cry with their technical level of detail - ask why they are being written this way - or better yet - write them yourself. If the testing is being taken away from you completely, don't let them. Don't let anyone tell you that integration tests are the same thing as acceptance tests. Coders, if you want your business owners to engage more, realize that giving them pages of cryptic, coder centric tests to review does not really encourage their engagement. As stated before, the idea of BDD was to simplify acceptance gathering so that coders would know where to start. It supports Agility by giving a venue for business people and programers to collaborate frequently. In the end, the slight reduction in speed along a more prudently chosen path will likely save you way more time than a radical course change much later in the project.

All of this raises a number of other questions I have been pondering for years - such as having different types of 'Customers', What sufficiency of testing levels for a given project, optimizing testing approaches... 

It also makes me hungry...

* - my wife wanted me to note that our household is vegetarian and that I am only promoting Steak in the 'software' sense. Thank you dear! :-)

Shortcuts don't always get you where you set out for

I have lived in a lot of places over the years. I have learned with each new place to get out and drive (or better - bike or walk) to start to get a feel for the place. Assess the lay of the land if you will. A number of years ago I was enrolled in a three year program at St. Louis University. So, I got myself a map which I kept posted near the door to my then apartment. Every weekend I would pick a place or a route to explore. I would mark it on the map either upon my departure or return. One day I had decided to check out some places downtown. On my way out, I asked one of my friends who was a native if they could tell me the best way to get to the first building. I thanked him and started on my drive. About three-quarters of the way there, battling down-town traffic I caught a glimpse of the building I was trying to get to down a quiet side street. This is nuts I thought. Why am I dealing with all this traffic when the building is RIGHT there! Needless to say, I promptly turned down the almost empty side street and headed merrily towards my intended destination. I remember congratulating myself that I had found a faster way than my native friend and making plans to tell him about the route when I got back. Then I made it over the first hill. While I could clearly see the building I had set out for from the initial route, what I had not seen was that this street ended with a very solid structure between my car and the streets beyond which would actually take me there. Of course, it then took even longer to go back, re-merge with traffic and get to my final destination. It turns out that St. Louis has a lot of these deceitful "shortcuts". My friend had a good laugh when I got back. I realized that I had momentarily put the real goal at risk by prioritizing speed to the goal. In the end it took a lot longer.

Let me tell you another story. One of my passions for many years is the traditional martial art of fencing. After a while, I wound up teaching it myself and running a salle. Now, most people who start fencing have at best, romantic ideas about what they are getting into. Few have any other martial arts background when they come to the salle. Fewer still, if any, have trained their bodies in the way that is necessary to move in our art. Our job is to train them how not to be hit by someone intent on hitting them. One of the first lessons a student learns with sword in hand is their guards and their parries. Inevitably, their first parries are all really wide (a natural reaction when someone is coming at you with a sword!). We calmly explain that while they may have avoided that attack, what if it had been a feint? Their large motion has left them wide open in all the other lines of attack. We work with them over and over through drills to parry a precise way and precise distance. We also repeat over and over that the goal is not to be hit and that part of that is to be efficient enough to be able to deal with multiple changing attacks.

Now one of the unique parts of our salle is that we believe in having all levels train together. There is value for the novices to work with fencers who already have the basics down. Additionally, the advanced students never stop drilling the basics. Eventually, therefore, we will get the question: why do you make us parry that way when the more advanced fencers parry differently when they fence?
Of course when starting out, focusing on form is important because it is really teaching you something deeper: how to be efficient so as not to get hit. More advanced students know more ways to do this by manipulating time, distance, blade engagement, etc. A novice does not. But all the novice sees is that the advanced fencer is using a smaller parry or a larger parry than what they have been taught. The thing is, it takes a minimum of two years of training and constantly being told the mantras about not being hit, and fencing is about trained, logical, efficiency rather than reactive reflexes before it sinks in.

So what does all of this have to do with Agile Development?

A friend and fellow Agilist, Chris Bartling, tweeted about a post he saw the other day: http://t.co/jq531Il

The blogger, Jay Fields, makes an interesting point. The goal of paired programming seems to address this principle of the Agile Manifesto: "The most efficient and effective method of conveying information to and within a development team is face-to-face conversation." It also hits some of the other principles such as attention to technical excellence and of course the primary one of continually delivering valuable software (by quality checks and reduction of truck numbers). The thing is, no-where in the manifesto does it say that to achieve these goals you have to pair. Rather it trusts motivated, self reflective and self organizing teams to come up with good ways to meet these goals themselves. All of this I agree with. yet something still troubled me about the post.

Jay is like the native St. Louis resident, or the advanced fencer. He understands the deeper goals and has enough experience to be able to draw upon a myriad of techniques to achieve them. The thing is, it takes a minimum of two years of fencing training and constantly being told the mantras about not being hit, and fencing is about trained, logical, efficiency rather than reactive reflexes before it sinks in. Even after living years in a city, my knowledge of the most efficient routes will pale in comparison with someone who has spent their life there. Similarly, the principles of Agile development look great on paper... a team reads it and there is much head nodding and affirmative mumbling. Yet, they don't really sink in until you practice them. Jay mentions that he has been pairing for years before this recent project/position. Advanced fencers fence, novices practice techniques. Advanced Agilists are agile, teams new to Agility practice techniques to become more agile.

My concern with the post is that like so many novice fencers, teams new to Agility will just see Jay not doing a technique which they may not agree with or find particularly painful initially. They will ask, why should we do it if he isn't... can't we just ask each other questions when stuff comes up? Unfortunately, too often, new 'Agile' teams will fail to ask those questions. The rigor of stand-ups, TDD, pairing, even collocation, gets the team members into the habit of asking the right questions. It helps create a culture of communication and collaboration where it may not have existed before. Of course, simply doing these practices is not enough, and should never be dogmatically considered the 'only way to Agility' - a good coach will work with the team as individuals to nurture the new culture in subtle ways relevant to the team and individuals... but discarding a new practice without understanding the underlying goal or reason can wind up being as wasteful as my 'shortcut' in St. Louis - and probably a lot more costly.