Milwaukee Agile Meetup - Continuous Integration Panel Discussion Summary

Last week I had the pleasure of being on a panel discussion about Continuous Integration at the Milwaukee Agile Meetup. My fellow panelists were Carl Schrammel, Adam Black, Jason Duff, James Carpenter and Adam Johnson.

The discussion grew out of the group's growing interest in this area and challenges that individuals and teams attending the Meetup had expressed recently. I know several people who said they had wanted to attend, so I thought I would post a summary of the discussion.

Q: What does Continuous Integration mean to you? When were you first introduced to the concept and how did it change your perspective?
A: After the initial text book definition was given ("Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day." - Wikipedia), the group continued to express their take on CI.

• CI is a powerful feedback tool for both engineering as well as project management - reduces surprises due to integration and deployments and makes the ones which occur more manageable.
• The benefits of CI are best reaped via automation, but even manual CI can bring great improvement.
• CI can help build confidence in the product and the team - one always knows the state of the code at the end of the day.

The distinction between Continuous Integration and Continuous Delivery (CD) was also drawn... with Continuous Delivery being an advanced practice where a team is able to deploy changes to production with high frequency. 

Q: What tools are being used?

A: The standard list of revision repositories, build servers, testing frameworks, continuous integration servers, deployment automation were quickly run through - basically if you are interested in CI there is likely a set of tools readily available for your language/tech stack of choice. It was noticed, however, that there are software products and systems which are not necessarily easy to integrate automation or other CI tools too. Particularly these tend to be configuration based systems, or closed systems which perhaps try to manage revision control internally - preventing access or control by external ci/build/test setups. This led to the discussion that aside from tools, the other aspect of CI is the practices and behaviors. These behaviors are often harder to implement than the tools. However, if the behaviors are embraced by a team, even the most simplistic of CI tooling can become powerful. The further consensus of the panel was that it is best to start simply - implementing behaviors and supporting tooling gradually. In this way, each advancement should be lauded as a positive win in the right direction rather than disappointment at not having achieved an elusive ideal CI or CD state in one try.

Q: How to prove the value of CI to developers? to business/product owners? how soon can one expect to see value?

A: The answers to this echoed some of the previous discussion. The general feeling was to take each instance in a case-by-case basis and look for opportunities to introduce behaviors and tools pragmatically to provide greater insight into specific areas where a team may be experiencing pain. Examples: recurring bugs, painful massive code integrations, lost code, misunderstood and mis-implemented user requirements, painful coordination/integration with other systems/teams on deployment, out of sync versions of code between environments (Prod getting ahead of Dev or Test), not being able to do certain types of testing. All of these were brought up as pain points which some degree of CI could have alleviated. 

Since CI sits on the front lines of ever changing code, the positive impact of CI can be seen very quickly (as soon as one build cycle in some cases). The key is to communicate this to the parties who would most appreciate the benefit. This is where a CI server can help by providing dashboards and automated messaging, but even utilizing the low tech approach of holding stake-holder meetings before and after a CI 'event' can shed light on the benefit of the activity.

The panel agreed that in order to get the buy in to do more advanced levels of CI requires the building of trust, something familiar to Agile Coaches and leaders everywhere. However, even the most rudimentary CI practices can provide excellent data and feedback which, if communicated properly, can facilitate the building of that trust. Due to the fact that CI can generate data about the health of the deliverable every time a change is made, it can be one of the most powerful and prolific sources of data for information radiators. The trick as with all data is to mine it for the most relevant information to the message you wish to communicate.

Q: Who owns CI? Who sponsors/drives for CI?
A: The panelists seemed to generally agree that in most large project teams, there was a dedicated Operations team in charge of maintaining the CI tools (build servers, repositories, deployments and deployment automation, ci servers, etc). However, many expressed how this could lead to the negative effects of siloing. Panelists shared their experiences with teams who used various means to combat this. Rotation of members into the ops team, rotation of ops duties among teams, the use of embedded testers to serve as the bridge between dev and ops, and use of separate scrum-of-scrums to coordinate dev teams with ops.

We did not get too much into the sponsors for CI. I would suggest, as with trying to get buy in, any individuals or groups who feel they would get a benefit out of the use of CI could sponsor it. I have seen CI started as an engineering initiative and I have also seen it sponsored by Project Management and even Product Owners.

Q: Of those doing CD, has anyone automated the signoff process in tightly controlled organization/industry?
A: No one had seen full blown automated sign-off at the level of the typical product owner, upper management deployment committee. However, several had seen groups who had automated sign-off at lower levels, such as moderate UAT sign-off. This can help pave the way to make the manual sign-off at the executive level easier once trust has been established through a record of reliable test/dev/pass/sign-off cycles.

The evening flew by and that was about all we were able to get to. Prior to the discussion, David Neuman from Redpoint Technologies sent out a list of questions to the panelists that would serve to seed the discussion. I think we hit most of the items tangentially. If anyone would like to discuss them further, leave a comment or email me.

The last question on the list asked if there was one recommendation we have for successfully implementing and leveraging CI. 

Build Automated Tests. Even if you still have to trigger your builds or other steps manually, having automated test suites that you can run against your built code is one of the most powerful tools to start getting an ROI on your CI efforts. Furthermore, many of the behaviors and practices of CI which are difficult to master initially revolve around disciplines of TDD. If you have automated tests, it is easier to start to instill these disciplines in your team. As some of the other panelists and I were discussion after the meetup, automated TDD is fun... it makes coding like a game with almost instant gratification to our problem/puzzle solving efforts. Making something fun can really make developing good habits to support it easier.

The language you code in could say a lot about your developer culture

In his essay on "Race and Language" Edward Augustus Freeman suggested that language, rather than race (genetic similarities), might be a surer indicator of cultural kinship among peoples. Professor Freeman wrote: "Every word that a man speaks is the result of a real, though doubtless unconscious, act of his free will. ... A man cannot, under any circumstances, choose his own skull; he may, under some circumstances, choose his own language. He must keep the skull which has been given him by his parents; he cannot, by any process of taking thought, determine what kind of skull he will hand on to his own children. But he may give up the use of the language which he has learned from his parents, and he may determine what language he will teach to his children." As such, the choice of language made individually and in the aggregate points to attributes of the language and those already speaking it which the new speaker of the language appreciates.

Perhaps this is why few topics among developers can start a 'religious war' faster than that of programming language.

The use of language is in essence a tool to capture an idea and communicate it to another. In IT we largely look at languages as tools to efficiently solve a problem. By 'solve' we really mean 'communicate an algorithm to the machine'. Just as some spoken languages are better for communicating facts or legal concepts and others are better for poetry or philosophical abstractions, each programming language is more efficient at 'solving' some technical problems than others. One may be excellent for modeling relationships, another for solving concurrency problems.

Looking at language this way is a good pragmatic approach to choosing 'the right language for the job', but it also can subtly mask or neglect other factors in language choice. Namely, the speaker and prospective audience plays a large part in what language communication is captured and how that communication is shaped. The above take on the use of programming language measures its efficiency in processes or cycles or memory consumed. It is the run-time efficiency - or how easily the computer understands and can execute the message. However, when using high-level programming languages, that is really misleading. It is really more of a measure of how good the compiler or interpreter for a given language is at translating the solution into efficient byte-code. So, compilers and interpreters aside, the real audience to gauge the efficiency of the language used is - humans.

Looking at how we use language it seems to lie upon a continuum between communicating to a large or a small audience. On one hand we have the language of media news reports, which linguists will often point to as a general epitome of the common speech of a people. It is intended to be understandable and consumed by the largest portion of a population. Prose is also near this end of the spectrum. On the other end you may find cryptography, where the audience is narrowed to perhaps only one other person. Sonnets, slang and technical jargon are closer to this end. They use conventions and specialized vocabulary which only a smaller subset of the population would have exposure to. Obviously a particular communication can lie closer to one end or the other given both the speaker and the audience. For instance a technical treatise might use highly exclusive vocabulary, but be written utilizing the most standard academic grammar and style. This 'regulated language' ensures that the message, while directed to technical specialists, will be understandable to the widest range of that population. It could even be accessed by outsiders armed with a good dictionary. Slang, on the other hand, often develops out the needs of the speaker... Complex grammar and punctuation is boiled down sometimes to only partial words. This makes the message easier and faster to say. To achieve this, the complexities of grammar and punctuation are replaced by context. One must understand this context, often cultural, to unpack the meaning behind this terse communication. It is particularly difficult for outsiders to master, and often intentionally so.

The intended range of audience has certain temporal effects as well. A loose jargon may make it easy for a small, tight knit group to communicate efficiently. However, it will be much harder to decode the meaning years later, perhaps even by members of that same community. Formal and standardized grammars and punctuation provide a degree of staying power to a language, allowing it to be understood for generations.
On can see this in development shops where one language is the exclusive language of development. On one hand a particular language may have been chosen for its similarity to human language. This may make translating business requirements into code easier. The same language may make it extremely difficult to describe the algorithm for performing a particular calculation. With greater complexity, it makes it more likely that a mistake will be introduced. The wordiness of this complexity can also make it harder to find the mistake. It also assumes that those writing code write well in their natural language. Run-on sentences can exist in code. A run-on piece of code is recognizable by loose cohesion, which is well known to negatively impact its brittleness and maintainability.

As a developer it is important to know the ways of expressing algorithms that is most natural to the language being developed in. This is called Idiomatic Code. However, a more seasoned Lead or Architect may realize that the eventual human viewers of the code may dictate different ways of expressing the same algorithm. This may even call for a completely different language to be used. Perhaps it makes sense to write code in one language, but to write the tests against that code in another. Testing in another language is often a good way to extend the audience of code written to an otherwise smaller audience.

This also brings up the impact of language on developer culture. Code written in a language which is only understood by one group within a team can lead to a closed culture. Think of those developers who almost have a visceral reaction to coding in anything but their favorite language. This language is usually that of a closed culture. Entire myths can be constructed about the superiority of both the language and those who code in it. Ultimately, this can threaten to alienate the developers from others on their team (testers, BA's, designers) as well as the development community at large. Uncle Bob Martin touched upon this in his 2009 talk "What killed Smalltalk could kill Ruby too".

Then there is the case where the language is chosen for you. The experience in this scenario can range from mind-expanding to dehumanizing. Surely the Junior Developer can learn much from being made to code in another language, particularly if that language has been chosen by a group of seasoned veterans across an appropriate section of the team. In this case the developer may be introduced into whole new ways of communicating and problem solving. On the other hand, there is the case where the language is imposed by a corporate decision made years earlier which may have little bearing upon the way your team works today or the problems it is being asked to solve now. This is the fallacy of simply 'buying' a language as a tool among equal tools. When that happens, the choice can become less about the effectiveness of the 'tool' and more about the packaging/marketing (e.g. - IDE's, plugins, reporting tools, support plans, name recognition, third-party-apis and frameworks, etc.). In these cases, the developers may often feel as if they are being forced to use an inferior tool for what seems like an arbitrary reason.

Finding pragmatic ways to introduce other languages where appropriate can mitigate some of the negative cultural impacts and keep communication flowing. It also can make the developers 'smarter' by making them look at and 'speak' about the problem using different paradigms than they are used to. This is much the same as people who speak multiple languages. Becoming a polyglot programmer may just make you a better designer or architect as your pool of ways of looking at a problem will be larger.

What other ways does the language you code in impact your team or say something about you?