Milwaukee Agile Meetup - Continuous Integration Panel Discussion Summary

Last week I had the pleasure of being on a panel discussion about Continuous Integration at the Milwaukee Agile Meetup. My fellow panelists were Carl Schrammel, Adam Black, Jason Duff, James Carpenter and Adam Johnson.

The discussion grew out of the group's growing interest in this area and challenges that individuals and teams attending the Meetup had expressed recently. I know several people who said they had wanted to attend, so I thought I would post a summary of the discussion.

Q: What does Continuous Integration mean to you? When were you first introduced to the concept and how did it change your perspective?
A: After the initial text book definition was given ("Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day." - Wikipedia), the group continued to express their take on CI.

• CI is a powerful feedback tool for both engineering as well as project management - reduces surprises due to integration and deployments and makes the ones which occur more manageable.
• The benefits of CI are best reaped via automation, but even manual CI can bring great improvement.
• CI can help build confidence in the product and the team - one always knows the state of the code at the end of the day.

The distinction between Continuous Integration and Continuous Delivery (CD) was also drawn... with Continuous Delivery being an advanced practice where a team is able to deploy changes to production with high frequency. 

Q: What tools are being used?

A: The standard list of revision repositories, build servers, testing frameworks, continuous integration servers, deployment automation were quickly run through - basically if you are interested in CI there is likely a set of tools readily available for your language/tech stack of choice. It was noticed, however, that there are software products and systems which are not necessarily easy to integrate automation or other CI tools too. Particularly these tend to be configuration based systems, or closed systems which perhaps try to manage revision control internally - preventing access or control by external ci/build/test setups. This led to the discussion that aside from tools, the other aspect of CI is the practices and behaviors. These behaviors are often harder to implement than the tools. However, if the behaviors are embraced by a team, even the most simplistic of CI tooling can become powerful. The further consensus of the panel was that it is best to start simply - implementing behaviors and supporting tooling gradually. In this way, each advancement should be lauded as a positive win in the right direction rather than disappointment at not having achieved an elusive ideal CI or CD state in one try.

Q: How to prove the value of CI to developers? to business/product owners? how soon can one expect to see value?

A: The answers to this echoed some of the previous discussion. The general feeling was to take each instance in a case-by-case basis and look for opportunities to introduce behaviors and tools pragmatically to provide greater insight into specific areas where a team may be experiencing pain. Examples: recurring bugs, painful massive code integrations, lost code, misunderstood and mis-implemented user requirements, painful coordination/integration with other systems/teams on deployment, out of sync versions of code between environments (Prod getting ahead of Dev or Test), not being able to do certain types of testing. All of these were brought up as pain points which some degree of CI could have alleviated. 

Since CI sits on the front lines of ever changing code, the positive impact of CI can be seen very quickly (as soon as one build cycle in some cases). The key is to communicate this to the parties who would most appreciate the benefit. This is where a CI server can help by providing dashboards and automated messaging, but even utilizing the low tech approach of holding stake-holder meetings before and after a CI 'event' can shed light on the benefit of the activity.

The panel agreed that in order to get the buy in to do more advanced levels of CI requires the building of trust, something familiar to Agile Coaches and leaders everywhere. However, even the most rudimentary CI practices can provide excellent data and feedback which, if communicated properly, can facilitate the building of that trust. Due to the fact that CI can generate data about the health of the deliverable every time a change is made, it can be one of the most powerful and prolific sources of data for information radiators. The trick as with all data is to mine it for the most relevant information to the message you wish to communicate.

Q: Who owns CI? Who sponsors/drives for CI?
A: The panelists seemed to generally agree that in most large project teams, there was a dedicated Operations team in charge of maintaining the CI tools (build servers, repositories, deployments and deployment automation, ci servers, etc). However, many expressed how this could lead to the negative effects of siloing. Panelists shared their experiences with teams who used various means to combat this. Rotation of members into the ops team, rotation of ops duties among teams, the use of embedded testers to serve as the bridge between dev and ops, and use of separate scrum-of-scrums to coordinate dev teams with ops.

We did not get too much into the sponsors for CI. I would suggest, as with trying to get buy in, any individuals or groups who feel they would get a benefit out of the use of CI could sponsor it. I have seen CI started as an engineering initiative and I have also seen it sponsored by Project Management and even Product Owners.

Q: Of those doing CD, has anyone automated the signoff process in tightly controlled organization/industry?
A: No one had seen full blown automated sign-off at the level of the typical product owner, upper management deployment committee. However, several had seen groups who had automated sign-off at lower levels, such as moderate UAT sign-off. This can help pave the way to make the manual sign-off at the executive level easier once trust has been established through a record of reliable test/dev/pass/sign-off cycles.

The evening flew by and that was about all we were able to get to. Prior to the discussion, David Neuman from Redpoint Technologies sent out a list of questions to the panelists that would serve to seed the discussion. I think we hit most of the items tangentially. If anyone would like to discuss them further, leave a comment or email me.

The last question on the list asked if there was one recommendation we have for successfully implementing and leveraging CI. 

Build Automated Tests. Even if you still have to trigger your builds or other steps manually, having automated test suites that you can run against your built code is one of the most powerful tools to start getting an ROI on your CI efforts. Furthermore, many of the behaviors and practices of CI which are difficult to master initially revolve around disciplines of TDD. If you have automated tests, it is easier to start to instill these disciplines in your team. As some of the other panelists and I were discussion after the meetup, automated TDD is fun... it makes coding like a game with almost instant gratification to our problem/puzzle solving efforts. Making something fun can really make developing good habits to support it easier.

Aristotelian Software Development = Agile Software Development?

I admit it. I am an Aristotelian. What I mean by that is that of the three predominant ethical theories, deontology, utilitarianism and virtue oriented ethics, the latter makes the most sense to me. It is the ethical theory which was promulgated by Aristotle. Jon Dahl gave an excellent introduction to ethical theory and comparison to software development. Initially, he alluded to the fact that determining what makes a good programmer has less to do with external signs, and more to do with the programmers habits. He briefly makes ethical comparisons of software development methodologies. In this sense, he starts to look at the human element of software development. The rest of the talk, however, makes comparisons between ethical theories and specific programming languages. Perhaps this is because he was at a Ruby conference. Ultimately, I think that a closer analogy can be drawn between specific programming languages and philosophical principles from Metaphysics or Ontology rather than Ethics. This is because languages are tools for expressing concepts. Ethics however, deals with humans and human society. The goal of Virtue Oriented Ethics is for the individual to achieve a state of Eudaemonia: human flourishing. Applied to software development we could say we wish the team or project to reach a state of flourishing, or Software Development Eudaimonia. As such, I think that comparison to ethical theories is more appropriate to things such as Software Development Methodologies or Project Management.

I am an Agilist (no surprise there). If you have ever had the good fortune (Eudaimonia is also sometimes translated as good fortune) to be on a highly Agile team, you probably have a sense of this flourishing. Notice how I used the comparative when describing the Agile team. Agility is not a binary state as much as it is a spectrum. Similarly, Eudaemonia can be achieved to varying degrees. The recognition of underlying rules or laws which help a society achieve such a state is codified as Natural Law. In so far that Agile methodologies work because of their recognition of humans at the core of software development, I think Agile comes close to describing a Natural Law of software development. Yet it is one thing to state a Law or set of Laws and another to develop a lawful society. Dahl brings this up when he talks about how the XP rules as they are promulgated are very deontological. Following the rules does not ensure that you will be a good programmer. This is interesting in that the people who came up with XP were good programmers. The generation of their rules probably was a long process of self reflection and attempts to make both their code and themselves better. So while the rules they promulgate may be used deontologically, the rules  themselves were likely developed via a more virtue oriented or utilitarian analysis. A common concern is how to get a team or company to 'become Agile'. Downloading lists of rules is easy, transforming a culture is not. Approaching Agility as attainable by degree seems to lead to better transformations. However these transformations are not simply replacing one deontological set of rules for another. People sense a real change in how they interact.

Therefore I posit that Agility is the Eudaimonia of Software Development. To follow the Aristotelian thought that virtue is taught by role models, perhaps a better use of the XP rules is as good models to others; sort of a goal state for developers working on their own virtue. Yet this presupposes that there are virtues inherent to achieving Agility. What are these virtues? Where would we look to find these virtues?

In his blog, Todd Hoff compares Agile as Aristotelian with Waterfall which he deems Machiavellian. In this, the core element which he recognizes is a scale between Trust and Fear. He presents the Waterfall world as being fear driven, whereas the Agile world requires teams built on trust. I think this a good place to start with analyzing Agile from a Virtue Oriented perspective. In many lists of Virtues, the concept of Trust is apparent. It is usually listed as Faith. In our modern minds, this word often is equated with a naive, unfounded belief. The way we use the word faith throughout our language indicates trust born from an observation of habitual behavior. We can say a dog is faithful, or I have faith in my friends, team members or boss. We say we are faithful to our spouses. This is language of Trust. In many of the classical virtue lists, Trust/Faith is a core virtue. This is similar to what we see in Agile transformations. The failure of many teams to transform to a higher state of Agility is often tracked back to embedded distrust within the culture.

In order to place the virtue of Trust in an Aristotelian context, it should be on a spectrum, with the virtue found at the mean of two extremes. So let us say that on one side we have distrust breeding fear. On the other side we have naive or blind trust, where we have no reason to really trust. The mean then is a Trustful state - where the team members both work to be Trustworthy/Faithful and based upon the actions of other team members they Trust or have Faith in them.

Naive Trust

Trust/Trustworthiness

Distrust/Fear

What then are some other Virtues embedded in Agile which help to promote Software Development Eudaemonia? The XP rules mentioned by Dahl have been noted as being somewhat limited in scope. So,while they may be a good guide for some of the team, they may not be the best model for general Agility. In their broad approach, the Agile Manifesto and Agile Principles are a better suited for guiding general virtuous development toward Agility. In my next posting(s) on this topic, I will go through these documents and highlight traditional virtues which are embedded in the behavior they are trying to model. I will then discuss how these behaviors and particular techniques from various Agile methodologies can help a team or individual increase specific virtues and achieve greater Agility.