Agile Revisited

Dan North

Recorded at GOTO 2015

let's go ahead look at the 1990s that
1990s we're building software and we're
building software big ok people was the
theme of the nineties he was writing
topper in the knight ISM he was at
myself out there for a quarter of e50 he
was at school in the nineties
most of you he was born in them ok this
is this is your parents agile but it's
hundreds of people multi-year projects
you know if you if you were one of the
leading edge think companies you might
be a planning or at least every eighteen
months to get ahead of the curve and
these are multi-million dollar
investments okay so because they're
multi-million dollar investments people
are very serious about deciding what
they're gonna invest in what they're
gonna pay money for mmm so what does
that mean you have functional silos the
lots of handoffs you've got analysts
developers architects designers you've
got systems analysts business analysts
solutions architects technical architect
lots and lots of specialized roles and
huge amounts that handles between those
roles often those rules weren't even in
the same part of the company or the same
time zone okay
since was quite big and you had slow
fragmented technology okay so well I
mean we're slow I was working on this is
late nineties so this is kind of towards
the end of the nineties a reasonably
large billing system for a telco and I
was in C++ was quite a lot of it and I
might change a couple of lines of code
and then make and that was the morning
dawn right so compiling was reason to be
fast to take anything at 20 minutes or
less but then linking that's two or
three hours later right big application
and you deploy that and then you start
running out of memory on hp-ux box is
another fragmented technologies that
every single there were many unix's
right in our in our in Linux is kind of
one right there were many unix's every
major Tim manufacturer had a
corresponding Unix that was differently
broken differently weird okay
so IBM HP Sun which was consumed by
Oracle there was go there was over
dozens of these things okay from two
great parents of BSD and system five and
so and there were lots of slight
incompatibilities the there one of the
reasons that British Stallman brought
GCC apart from free software because
he's pretty militant about that was all
of the other compilers were rubbish and
so essentially it ended up with that
you'd use the Sun compiler was just the
bootstrap compiler for GCC so as long as
it could do that it was fine and a lot
of this attack the technology was
proprietary so you had to objective-c I
would argue is a much nicer object style
of C and C++ why does he plus plus win
because Objective C was a commercial
product we had to pay for it so so it
was slow fragmented technology and our
software development process was modeled
on civil engineering why is it modeled
metaphors right we're making this stuff
up in the 90s we're still making this up
who's got a parent in IT right I'm
seeing more and more hands each year as
I asked this today there are four hands
out of about 300 people ok literally
we're making this up this is not genuine
this is if there was an architect's
conference or a healthcare conference
most of us would have a parent who are
in it so so we model based on what we on
what we already know so we see
engineering and we think that's quite
rigorous I mean that's probably quite a
good thing we should maybe model
ourselves on civil engineering let's see
how that works so what is a civil
engineering at like civil engineers have
learnt that the later in the process you
get something wrong the more expensive
is if I discover that the elevator
shafts are in the wrong place upon a
giraffe's board I just rub it out okay
if they're in the wrong place once I've
sunk the pilings blimey
someone's going to be upset
likewise if I'm building two halves of a
bridge and they don't quite lis it can
be a really expensive day but it only
signed up with two bridges which is nice
so exponential cost of errors we know
this we've measured this we've all been
burned by this as civil engineers okay
so we're trying to minimize likelihood
of errors okay and the way we minimize
likelihood of errors is we front-load
everything we front-load all of the
thinking of the spatial stuff onto paper
we front-load
all of the planning of resources there's
a cement I'm going to need the sand I'm
going to need the the gravel I'm gonna
need aggregates all of the materials are
steel I figure out how much of that I
want that is an entire discipline in its
own right
it's called quantities of a quantity so
that comes along and estimates using
charts and things and whatever else
books exactly the right amount of parts
of stuff a project manager comes and
manages the project because if you're
going to have 500 electricians waiting
around because 500 bricklayers haven't
finished yet that's a really expensive
bit of climbing okay so the scheduling
matters when you're building hospitals
and so we have assurance through formal
sign offs is the architecture sufficient
well the architects think so but often
architecture is vanity architecture so
they're put in a tower or a pillar or up
here or something that's going to look
cool and so what are they get a
structural engineer lives in says night
no you're going to need a supporting
thing there and that's not going to hold
its own weight okay and so you would
trait on that and so the structural
engineer signs off the safety of the
building why because they if part of it
collapses we can go and see him or her
and we don't get in trouble okay
so that's the formal sign offs what does
that mean that means any plan we have is
very intolerant of slippage is this
sound familiar is this sounding at all
familiar okay so any plan we have is a
tolerant of slippage and handoffs then
become detailed become expensive and a
lot of cases become litigious right I
want double signatures and witnesses
that kind of stuff and so this is what
we learn and so as software engineers we
inherit this and we make a bunch of
assumptions if you assume that those
first three things you end up with those
two things in other words if I assume
that I need to front-load all of the
work and that the work gets
exponentially bigger as I go along then
if I make a mistake guess what I've got
a huge amount of collateral already that
I've got to go back and revisit and so
you end up with a reinforcing loop right
a vicious circle and so this was the
context in which a bunch of middle-aged
white men met in a cabin in Utah in 2001
so Bob Martin uncle Bob and he's quite
cross about some of these other stuff
and it turns out a whole bunch of people
quite cross about some of this stuff and
they have been during the night who's
quietly getting on with business quietly
delivering systems in big companies in a
very different way they've said well
what if it wasn't like civil engineering
and so they wanted a name for the
different thing so and Bob Martin's very
smart guy he was very deliberate he said
I'm only going to invite people who have
a proven track record of software
delivery this isn't going to be a
speculative hand-wavy thing this is
other people and he looks around the
industry and he sees scrum dsdm crystal
FDD adaptive and XP all of these
different methods probably six seven
independently racked up a bunch of
successes he looks across his peer group
and he sees people like the pragmatic
programmers Andy Hunt and Dave Thomas
and he gets all these people in a room
of stuff in common what we need is a
name we need a name for this thing so we
want to come up with a word of brand
that's going to represent what we value
and they came up with a word I'm sorry
that's not the way they came up with
Heather's so though if any of you guys
seen a demeans Heather's
there's wonderful moment in it where
on that's not the word this is the word
they're at with agile they said we will
be we will describe the thing in a fact
agile wasn't their first choice their
first choice was the word adaptive we
want something that adapt to new him to
in new information unfortunately one of
the guys in the room Jim Highsmith
already had a successful method software
delivery method called adaptive software
engineering which is based on ran and
and funny enough he didn't put his foot
down a site you can't call it adaptive
the other guys in the room said that
would be unfair we don't want to hijack
your brand we'll go with a different
word so our child was second choice so
the main output of the Snowbird summit
as it's known in 2001 was a thing called
the outdoor manifesto now there's only
thing you can probably get what I'm
going to say next the things I'm going
to talk about next is what my net I
Drive alleys no no I'm not and here's
here's why I'm not the ventilator here's
why I'm not is people tend not to look
past the default so you look at the
agile manifesto and it's a lovely
picture and it's got these guys all
standing by the board it's all nicely
pixelated oh and then you have you know
people have a process blah blah and then
you look beyond default because those
things are value statements in fact
they're examples of value statements
it's really hard to go and enact value
statements okay
so especially if you're learning a new
thing so if someone says a tennis coach
says okay so what you need to do is
focus on your foot position and keep
your balance slightly sensor and
slightly forward and lean into the ball
you're like what how do I hold the stick
thing yeah and so when you get a bunch
of value statements it's really hard to
work with those so it turns out when you
turn the page there's also a bunch of
what they call principles and there's a
dozen of these principles and only very
quickly go through some of these things
so you've got the idea
and continuous delivery of software
that's one of them and you want to
welcome changing requirements okay
these are things I can do I can have a
change in a climate like and welcome it
or I can push it away okay that's a
behavior I can adopt what else know I
can deliver frequently I kind of think I
could do it I could try and deliver
frequently as opposed to not frequently
so each of these things is comparative I
could do this versus not doing this and
some of these things are so obvious and
it kind of sets the scene about how bad
the industry was that someone had to
call these things out okay and if I had
a group of people at the front of their
discipline had to call these things out
there's some more things working
software as a measure of progress have
you shipped yet is it working can I use
it right we'll call that progress
self-organizing teams self-organizing
teams means you bring all the different
disciplines together rather than having
very rigorous rigid handoffs well so we
go technical excellence Oh technical
excellence that would be nice right why
do you need to call out technical
excellence in software engineering I
don't know I do know I was there I was
writing some of that stuff okay and they
say things like on businesses and
business people and developers working
together we want the idea of building
projects around individuals around
people who really care about that thing
rather than you know we've got these
people that's just three worked out
there she'll tell you do the work we
want to value face-to-face communication
that's a big deal
simplicity all our say about simplicity
is simplicity it turns out you can spend
days talking about simplicity it's a
very very deep and complex topic
we're simplicity is a thing we value
right sustainable pace okay.we is
pulling this off once is easy you just
burn a bunch of people out okay doing
this sustainably again in the game and I
love because this is a room for the
programmers don't forget they said not
just sustainable for us
sustainable business people users
everyone else okay so it's not-it's not
enough for us to be sustainable if we're
learning everyone else out on the way
through and finally they said we need to
and tune on a regular basis okay so that
seems like a quite a sensible bunch of
things to want to do and this doe
forgets this is in the 90s okay so so
then we've got this brand
whoever D of a brand so what's it brand
then a brand in technical language a
brand is our to brand something is to
give a product a distinctive identity
okay so what that means in linguistics
terms is a thing there's a field of
linguistics called semiotic semiotics is
the Association of meaning to symbols
what that means in this context is it's
a brand it doesn't matter what the word
is it matters what your association with
the word is what does the symbol of that
word cause you to feel okay so if you
hear I know so so product brands their
product brands that make you feel safe
right Waitrose john lewis we think of
those as trustworthy brands okay little
Asda we think of those as maybe sort of
towards the the less expensive end of
supermarket kind of brands so there's
there's an association with brand and
there was a brand that came out of this
whole period that kind of one okay and
it's a brand called scrum so the brand
of agile kind of one the brand of agile
stuff it's just that it wasn't agile it
was scrum or rather the two words have
kind of become synonymous because scrum
doesn't represent these things at all it
represents these things okay so it
technical excellence kind of fell
through the cracks a bit or at least
strong doesn't have an opinion about it
simplicity again scrum doesn't have an
opinion about that it does however have
an opinion about face-to-face
communication usually in the form of do
you have the estimates I are you
committed to them let me beat you up
okay where's my stuff delivering through
the I'm writing scrum hard here on it
scrum changed our industry okay
20 15 20 years ago people were
delivering in years and thinking that
was normal scrum caused people to be
able to do
in months okay now that's kind of my
point is that was true in the nineties
let's take a look at scrum now this is a
small screen grab from an article or
info q yesterday okay
leading edge technology site they have
an article there about Capital One's
there they go big bank using agile
methods and well they say they say we've
but we've trained three thousand people
in the agile methodology right because
that isn't a thing there isn't one
there's several of them but what we've
done is we conflate the word agile scrum
we say we're using the agile methodology
and that's kind of normal now okay so
things like so when you talk about we
don't talk about custom anymore that
it's always product owner ago product
owner is actually a proxy into the
business it's very rarely an actual
used to be a project manager and didn't
quite get all that scrum master nonsan
this and wants to be able to shout at
people still okay they really don't want
to let go of that so in a product owner
okay and and sometimes I have domain
knowledge and sometimes they don't and
then to the breaks so in the 2010 thing
so in our decade now well we've got
smaller projects that's pretty cool
we've got significantly smaller projects
now we batch money in smaller lumps and
because people like the message of agile
they like the idea of better faster
cheaper and all this kind of stuff we
tend to have cross-functional feature
teams now I here's ten feature teams a
lot either scrum teams or feature teams
so there is this idea now that there is
a generic grouping of human beings which
is a formula of n developers Plus M
testers where n is strictly less than n
plus one or slash a fraction of a be a
okay and that's the unit of delivery and
that's software soft okay we're done
we've all go home now so we have these
cross-functional feature teams and what
else do we have
we've got faster technology we've got
commodity technology okay so I don't
need to think about which variant of
Unix I'm using now okay I can an awful
lot of stuff I have your local source I
have a small manageable process okay
that doesn't take reams and reams and
shelves of documents to to define my
process so right they're doing scrum
who's new scrum or some variant of scrum
okay who's doing Kanban or some variant
of Kanban okay keep your hands up hands
up keep your hands up if you're limiting
working process measuring cumulative
flow using that to make predictions
about SaaS that you're not doing
campaign right what you're doing is
scrum without any of the scrum yeah
you've got fed up this from so you threw
it on the governance and now you call
that process you're doing unmanaged
delivery awesome right
I'm go read up on Kanban it's quite
useful site and but we've got a
processors modeled now an iterative
delivery we're trying to ship thing this
turret of Lee so um it always sounds
pretty good except there's an elephant
in the room here except we're still
batching the money so you still have
programs and program definitions and
annual budget cycles and even though
I've got my scrum teams and I might have
several scrum teams I'm planning a I
know millions typically or pounds or
spend across a bunch of people for a bit
of time okay so the money is batched in
that conversation all those
conversations and often a laundry list
of what I'm going to get for that money
that all happened when outside world
before delivery downstream is not
batching of infrastructure okay I have
two large organizations I've been
working with in the last 12 months both
of whom essentially would love to on the
delivery function own
their own infrastructure and they don't
and each of them spend inordinate
amounts of mental anguish and actual
physical time and more mean things than
any human being should be subjected to
to try and get stuff into production
okay so there's this scrum in the middle
and then just for Adrienne we have water
scrum full okay so the idea is that you
have water for either side of scrum what
batching a bit of spin around is that
model is nothing moves faster because
the lead time is from there so there and
if there's a bunch of time there and
handoffs and a bunch of time they're at
hands of people could care less what
we're doing that okay we care because it
means it's more it's nicer way to work
so then we're next with the agile so
this is it's a cloud and it's a question
mark do you like what I did there it's a
cloud and it's a question mark you're
welcome oh so so this whole thing about
cross-functional feature teams where did
that come from someone said there should
be cross-functional feature teams rather
than people working in functional silos
there's a lean principle so one of the
core ideas in lean operations is that
you move the people to the work and it
doesn't sound that revolutionary
until you realize that in almost every
industry particularly manufacturing
industries you move the work to the
people you have people at workstations
River they are and you take the work
item and I do some stuff to it and then
so this lovely chap Taichi I know in the
1950s said that's bonkers there's no
value being added to the work when you
transport it yeah I need to find a whole
bunch of things where there's no value
being added to them when you do things
so there's no value being added when you
yourself walk around if I take this
bottle over here right now here I can
get this lid over here right no value
was added during that
okay if I produce too much of a thing
more than anyone wants I've just
produced too much of a thing there's no
value I did that so transporting work
around is a non value-adding activity
and with knowledge work that's actually
transporting development of stuff around
so what how do you counter that you
bring all the people to one place okay
that makes sense so then you get
cross-functional teams however when you
pan back and start doing larger programs
that work you've kind of got the same
problem again because now I've
predefined my set of scrum teams or
feature teams and they're doing what
they do and I've got this big program of
work I need to figure out how to slice
up the work and funnel it into all of
those feature tapes so I need to do the
kind of the divide piece there and then
the joining piece at the back of anyone
who's written any kind of fork/join
scheduling semaphore I'm looking at you
Farley right this is insanely difficult
okay and you're always moving at the
speed of the slowest piece and so you
end up with that problem and so what we
want to do we want to move the people to
the work in other words what we want is
a much more fluid organization so now
you start thinking at team scale is the
group of people to solve the problem if
the problem I have is a big wide problem
I need a big wide group of people that's
my team my 50 60 70 hundred people
pointing out a problem that's my
definition of team I now figure out or
ideally they figure out what's the best
way for those guys to divide themselves
up such as they can face off to the work
coming down the pipe
since the work coming down the pipe is
going to change as a function of time
the org structure the team structure is
likely to change as a function of time
given that the types of work I'm
engaging in are going to be different
types of work it's likely the different
groups of people are going to be best
suited to tackle that work and so now
you start to define the groupings of
people as a function of the pull on the
okay the types of work that we're trying
to do in order to move the business
forward and suddenly it's a less trivial
problem than n plus n plus
tester okay it's a little bit more
sophisticated than that but remember
they are actual human beings here okay
so it's not just put it in a bag and
shake them up these people have
aspirations and careers and families and
hopes and that kind of stuff
okay so let's be sensitive to that so we
want more than no process I'm looking at
you can that right well more than no
process but we don't want rigid process
okay so we've swung the pendulum both
ways now okay rad in the early 90s was
look they were done with all this
nonsense and then they're kind of
heavyweight if you like scrum stuff what
we're doing now is kind of a pendulum
the other way somewhere in the middle is
a same grouping of things we could do so
now what does a team start to look like
a scale okay let's go so further that's
we do now we can start measuring
business impact what all of the software
metrics we have all the metrics we have
from things like scrum even XP all these
things they measure software
productivity okay how do you measure
software productivity we measure it in
terms of velocity okay
so features feature story points stories
there's a measure of units okay and then
we take this team scale measure of
velocity and we say hey this is how much
this is how fast we're going it's not
how fast we're going
it's it's how much distance we're
covering that tells me nothing about
whether I'm moving towards my goal how
soon are be at my goal or whether the
distance I'm traveling is even in a
sensible direction don't tell me any of
that I just told you I would deliver
this many features okay and why do we do
that Oh Dan you're beating up on these
things many people have been idiots they
weren't they were using the best
information they had they were using
proxy metrics we can now move fast
enough that we have real metrics okay I
can measure business impact because I
that I can immediately measure whether
it matters and if it matters I keep it
and if it doesn't matter I kill it and
in fact it's so inexpensive I can do
several of them at once okay and the
ones that win pay for the ones that
don't and it turns out if you're a
venture funding
that's basically your day job your day
jobs let's find a bunch of people who
might win and the ones that win pay for
the ones that didn't and everyone has a
great time so software productivity
isn't a thing okay
velocity isn't a thing so in terms of
software then less is more
right less is more like surgery I don't
want your stinking software
I don't want your stinking features what
as a trader I would like to be able to
trade with my mind okay
oh that's be able to trade at the speed
I can reason about the market rather
than having to click around on a bunch
of stuff okay
that's irritating right as a shopper I
would like to now be wearing the new
outfit that's what I want my goal is to
be wearing it in fact my goal is for my
friends to say that's a really nice new
outfit you've got that okay so what's
the lead time to that well clicking
around your stupid website isn't helping
me yeah so I don't want software I want
fewer features I want opinionated things
and I want them to be tiny so now and
it's gonna get me in trouble with engine
I'm gonna assume technology is instant
and free okay
you'll notice the asterisks there at
least compared to the 90s but it is now
instant and free what do I mean what I
mean is I used to have a six-month lead
time to get a server I can now spin up a
thousand servers in a second okay or
even better I can have a thousand
servers sitting up there and I can
deploy containers into them in literally
milliseconds this is bonkers this is
bonkers fast so technology is internet
free it's free and instance who writes
my programming languages are no longer
proprietary as someone who used to code
C++ for money right the fact that I can
type into my IDE Java code and have it
compiling behind my fingers as I type is
magic it's still magic right Gohn has a
slightly faster compiler than I can hit
enter these things are so writing code
is now the speed that I can think and
manipulate code it's not the
women are some tolling build again build
is near-instant if that's scaring you
use scarlet okay and then it'll build
much slower if that's scaring you
then then use maven to build scarlet and
all fine okay see if it's going too fast
those are the breaks okay I may say that
cuz Dean warm clothes downstairs
teaching a scarlet class though it's a
bill a provisioning provisioning is free
instant okay it's dot it's cent per day
right it cost me a few set a few US
cents a day to run a server it's bonkers
I can have thousands of servers and I
can put that on you know costing me
about cut the price of a coffee okay
deploy it again is free is instant all
of that stuff exists that infrastructure
as a service platform as a service and
then monitor is free and massively at
scale because Adrian okay because Adrian
built me a massive monitoring and
alerting capability that as a side
effect streams video he says stealing
his line so so now I want to embrace
continuous delivery what does that mean
two weeks is an illusion
reference everyone recognize it right
planning having a planning horizon of
two weeks is nuts if I can try five
product IDs in the morning okay it's
both too short and too long I should be
having an opinion two or three months
out I should be having an opinion two or
three hours out anything in between is
random outcomes create options so I've
got an emergent system here so it's not
about changing requirements its
discovery of requirements through
delivery it's a complex adaptive system
I'm now got a rolling up X versus and
upfront cap X I don't need to think
about dumping a bunch of money in I can
just rent my service as I go and when
I'm not getting value anymore I switch
it off or a dial it down
and finally our investment collaboration
over commitment it's still there but now
we're having converged collaborative
conversations about investment should we
invest in this thing rather than give me
an estimate hold you to it
would you use a 1990s computer now to do
stuff other than for retro right no and
why would you use in 1990s methodology
okay so here's our manifesto again right
I think they made a pretty good joke
that it's emerging requirements is what
I change business impact is what I
change business and voters and everyone
else build products around motivated to
continuous reflection deliver
continuously that wasn't bad for a first
draft right 15 years later the vast
majority of that is still relevant
we just need to adapt it differently to