Kicking the Complexity Habit

Dan North

Recorded at GOTO 2014

good morning it's technically it's good
afternoon
ten past twelve so good afternoon
there's a British thing that happens
that when the first time you see someone
in a day you say good morning it doesn't
matter what time of day it is and
there's a good morning it's kind of like
a greeting protocol good morning good
morning how are you I'm fine
you never ever say how you are okay
that's breaching protocol the problem
happens around about noon because you
know just after noon that it's not
morning anymore so now you're kind of
lying so you can actually you can use
this to spot British spies okay so what
you do is you say like what you stand
near them within I it within a line of
sight but not within speaking distance
really close to midday and if they're
British their palms will start to sweat
because like he's going to engage me in
a minute I'm gonna need to say good the
word Ono that so why on earth am I
talking about this nonsense because
because it's because it's a it's an
idiom it's a it's a model we use it's a
way we identify ourselves our
Britishness us British folks now we have
these funny little quirks Americans have
their funny little quirks as well okay
we're gonna be looking at some of those
sort so don't you bear in mind these
sorts of things this talk is called
kicking the complexity habit okay so
kick in a complexity habit there's a
kind of it's a you can see it's like a
metaphor at work there is like the idea
that it's an addiction it's a habit
which is euphemism and and you look at
the title kicking the complexity habit
there's four words there and I'm
thinking they probably don't need to be
four words they're kicking complexity
says nothing less does it right still
kicking of a thing so it has the kind of
it's a habit and we're trying to break
the habit it also has a nice second
meaning which is like we're kicking
complexity so I quite like that even
that's a bit busy complexity okay why
why am I talking about this the title of
this slide because complexity is
everywhere unnecessary complexity is
not curious as the title of a talk
called kickin the complexity habit or as
something as surely harmless as greeting
someone okay
there's complexity in all these things
where I want to try to get you guys
today the thing I'm going to try and do
and almost certainly fail spectacularly
is to help you develop a sensibility
around complexity I want to make you
aware of complexity okay
how many I imagine most of you haven't
you've seen the video with the
basketball players and the bear kind of
wanders through and break-dances his way
through okay if you haven't seen this
Google for break dance have a look on
YouTube for break dancing bear it's
incredible there's literally a bear
breakdance is backwards through a bunch
of people playing basketball and you
don't notice the first time you watch it
and the second time you watch it anyway
I won't tell you that the second spoiler
um the point about this is once you're
aware of it you can't not see it and I
want to kind of tear away the veil a bit
so I'm going to try and do okay so i
i've been i watched a video not so long
ago by a guy called brett vector called
inventing on principle
use our reference book so they don't
seem to be referencing videos go and
watch inventing on principle it's
wonderful it's a wonderful wonderful
talk about the idea that there are
certain people kind of a bit evangelical
kind of a bit hand-wavy kind of a bit
loud and outspoken who want to change
the world in some very narrow way okay
so Brett Victor's principle is you
should get immediate feedback from the
thing you're creating it helps the
creative process so I was thinking about
this nice thinking do I have a principle
do I have any principles yeah as a
Groucho Marx as these are my principles
if you don't like them I have others
what would my guiding principle be for
this and and I've ran this I did the
wife test I ran this by my wife and I
said is this the kind of thing I say and
she said then you say this all the time
it really shouldn't be this difficult
and I will look at code I'll look at
design oh look at organizations I'll
look at teams I'll look at people look
at systems it
really Oh God it really shouldn't be
this difficult so what am I going to try
and talk about over the next bunch of
time um I want you to know your enemy
okay so there's kind of three three
parts of this talk for some value of
three the first thing is I want to help
us learn to identify complexity
complexity is everywhere and it is so
subtle and so insidious you don't notice
it okay
once I mention that the room that you're
in is full of little gold chairs with
little gold chair backs
you can't not see them now yeah you were
completely ignoring them you'd already
deleted them the people at the front
here the room isn't full of little gold
chairs little gold chair backs okay but
everyone else okay um so mostly what we
do is delete stuff okay deleting detail
deleting stuff is a way of that our
brains handle complexity the
overwhelmingness of complexity at
identifying and then okay so now we've
identified it so warm
okay identifying fouls - so what test so
what can I do then well reducing
complexity let's assume we are already
in this massive ugly horrible morass of
stuff we call complexity what can we do
to reduce it okay and then we start
doing that and that feels good and we
gradually bring in whatever the opposite
of complexity is and and then what we
want to do is as we move forward to stop
making such a mess
so the third part is how do we avoid
creating complexity in the first place
so that's what I'm going to try and go
okay so first off let's talk about
identifying complexity there are many
definitions of complexity what's
interesting is when as soon as you start
looking into fields that specialize in
studying things like complexity they
come up with technical jargon that
sounds really similar and means very
different things and then the way it's
like a club because then the guys from
the other club use the same words in
slightly different ways and then they
know they're not in the same club okay
so it's like a sort of password thing so
in systems thinking you have what they
call detail complexity and dynamic
complexity so detail complexity
is lots and lots and lots of bit many
many many bits and you have to learn all
the bit with sufficient study you can
sometimes learn all the bits dynamic
complexity is emergent behavior so
whether as a system is actually a pretty
simple system right weather has
temperature humidity air pressure that's
most of weather okay the problem is
emergent behavior
the problem is weather fronts and
different bubbles of air and
temperatures and humidity banging into
each other which means it is impossible
to predict weather particularly well we
were coming here to Chicago from London
we looked at four different websites for
the Chicago one week weather forecast
four completely different sets of
answers all four of which were wrong it
turns out because then this happened
which are quite like it's sunny outside
if you're watching at home and a parent
is going to be sunny all week which is
awesome so yeah so that's that's dynamic
complexity dynamic complexity you kind
of got to live with right emergent
behavior you kind of got to be able to
react to or try to react to this isn't
that what I'm looking at is detail
complexity is fine grained stuff in
called Kevin which is a Welsh word and
you have complex versus complicated okay
why doesn't that make it more
complicated or possibly complex I don't
know but you have these different
there's these things and so then when
the conniving guys in the system
thinking guys get together they'll fight
but it's sibling rivalry because they're
all trying to understand complexity
anyway that's not the definition of
complexity I want to go with the
definition of complexity I want to go
with comes from this man this is James
Lewis he's a he's actually a principal
consultant these days with thought works
which is all very fancy and it means so
the further you go the more kind of
principal you become as a consultant you
do less and less and charge more and
more for it's a fantastic model so he's
now charging the road for doing almost
nothing he's that good right but he came
up with my favorite complexity metric
and he said so he has an interestingly
large head okay so he said I don't like
to look at any code that doesn't fit in
my head that's bigger than my head
okay say his head about that big okay so
here here is some code on a screen this
screen is anyone using sublime text
editor few hands this is the original
blog post for the guy who was writing
sublime text he's like this is my world
I want to solve that okay so now look
we've got one two three screens another
two little half screens off to the side
there okay
let's apply it the James fits in my head
metric well patently that's too big to
fit in his head that that will fit in
his head okay however there's another
axis here whoa look at that that's three
that's going to require more than three
James heads to even be able to look at
that screen of code okay that is
insanely more complex than we want to be
looking at and this right there's two
there as well okay it's not really
physically what otherwise what we're
going to have to do if James's head
becomes our standard unit of complexity
we're gonna have to do is chop off
James's head and put it in a safe in
Paris okay in the SI Institute and
that's not gonna yeah and we'll get a
little bit damien hirst at that point so
not that but what are you saying is is
it's about he can only reason about
something if it fits in his head and
that made a lot of sense to me and that
got me into all that well into all the
rat holes on the trying tell you about
now he said so I can only reason about
something if it fits in my head and I
immediately went to the contrapositive
of that for any logicians which is this
if something doesn't fit in your head
you cannot reason about it it turns out
most of the stuff that we deal with
doesn't fit in our heads sizes of
systems are software organizations
product domains don't fit in our head so
we come up with ways to cope okay so it
doesn't fit in our head we can't reason
about it so what do we do instead we do
what we know we fall back on habits we
fall back on routines okay this I'm
going to warn you now the screens about
to get complicated I'm going to put up a
diagram but I'm going to try and explain
the diagram to you
in about three minutes so just assume
even if you if you've not seen anything
like this thing you're about to see
before this is from a area of research
called systems thinking and systems
thinking
the idea behind systems thinking is you
have the world is made up of
interrelated components and the
relationships between the components and
the loops that those relationships form
are far more interesting than the
components themselves okay so this is
beyond cause and effect to systems of
interaction and systems thinking has a
set of what are called archetypes which
is for the software guys it's kind of
like patterns so they say archetypes of
things that occur again and again and
there's an archetype that looks like
this
it's called the shifting the burden
archetype okay the way shifting the
burden works let me just explain the
duck let the symbols here this is a line
joining these two things okay so
anything in a box is either some kind of
measure something I can measure or it's
some kind of phenomenon it's a thing
okay
so and then what we have is connections
joining the things some of these
whenever they form a loop they form one
of three types of loop
they either form a balancing loop it's
it's unlike little scales they're drawn
up scales and so the idea there is over
time these things will balance out or
they form a reinforcing loop that's
supposed to be a picture of a snowball
rolling down a mountain gathering snow
and you know avalanche and all that the
third one that isn't on here is an
oscillating loop so sometimes it kind of
keeps swinging between two stable states
like you're standing in the shower and
you go is too hot it's too cold it's too
hot cool if the if the amount of time it
takes for the temperature to change is
just wrong you can be doing this for a
while okay whilst having the most
miserable shower so okay quick example
back injury so the back injury the
source of the problem is the root cause
at sorry back up
what is this diagram showing us this
diagram is showing us this shifting the
burden says for any given for any given
set of symptoms for some presenting
symptom there is typically some kind of
underlying cause to the
and some kind of symptom to that the
presenting symptom what we tend to do
excuse me no guess but what we tend to
do is our attack this loop here because
you see this link here is a media
something happens here immediate effect
here this bottom loop here is about
attacking the cause which looks like a
similar loop it goes round the circle
except here is where the magic happens
there's a delay any any system in which
there is delay we become reluctant to do
because there's uncertainty there and we
fear uncertainty and so we tend to go
for the easy for the quick win so that
the most common example of shifting the
burden back injury so back injury is my
under and is my underlying cause I don't
know I've got a back injury what I know
is I have back pain and I go all my back
saw oh so I think all what should I do
for a sore back what I do with this all
back I'll take some painkillers I'll
take painkillers there we go quick fix
right back pain back under control
that's fantastic what should I be doing
well what I should be doing is down here
exercise diet lifestyle physio all of
that stuff that's going to systemic lis
over time alleviate my back injury help
me recover and therefore remove the back
pain I don't do that though I do the
quick win I take the painkillers the
problem happens over here over here we
get an unexpected side effect okay an
unintended consequence of the
painkillers in this case what's the
unintended consequence of the
painkillers I start quite liking
me and painkillers have a special thing
going on and then not just those
painkillers because they don't really do
it anymore the other pain right and so
you get this whole thing going on but
also don't forget we've still got the
back injury that hasn't gone anywhere
and so now because I'm taking
painkillers my back injury gets worse
and worse and worse until it's
unrecoverable okay and so what happens
here now this is a reinforcing loop
again
so side of it I've become addicted to
painkillers that's bad okay that's bad
that means my back doesn't get any
better okay we're not here this is not a
medical conference so let's look at a
different example let's look at the
example where we have system complexity
big complex ugly codebase and I get
stuck in my codebase and I can't reason
about this code it doesn't fit in my
head okay but I'm not aware of big
system complexity that's too abstract of
a thing what I'm aware of is this I'm
where a frustration I'm aware of
confusion I find the code in which I
feel the most helpless of you know quite
a wide breadth of enterprise-e type code
that I get to look at is usually rails
apps I have no idea what's going on in a
rails app okay and if something goes
wrong in the database I have a 17 deep
stack trace of staph somewhere deep in
there in some cryptic ways I don't know
I don't know what's going on in there
there's magic I also can't look at code
and reason about it because some meta
programmer yeah right has injected a
bunch of crap that I can't see and left
no trace that they've injected a bunch
of crap so the code I'm looking at is
not the code that roams thanks right
this happens to me a lot I've never felt
more helpless than in someone else's
rails app okay that's just a personal
thing and I love Ruby Ruby's a great
language rails I don't understand it
apparently lots of people like it so
I'll leave it there so frustration
confusion I don't I can't reason about
this code I can't reason about what it
does where I should make the change I
want to make impact analysis how to
figure out whether the change I'm going
to make over here is going to cause a
whack-a-mole problem over here okay I
don't have those things available to me
so what do I do I go lalala it's a
really complicated codebase I work
around it
I typically I see this massive nested
wave Eve if nested thing going on and I
just squeeze my new condition in there a
little bit yeah I could tidy it up but
I'm kind of in a hurry and it's only one
little if condition oh and it's there as
well I went there and I forgot one over
there right ship it yeah cool say
nothing right so this is what we do what
should we be doing we should be tackling
the underlying complexity that's really
hard it's usually really hard usually
fairly thankless and also don't forget
this delay we've got a ship stuff and
what am i saying to the people yelling
for the stuff well I'm just kind of
taking care of business over here and
I'm doing some cleaning up and I'm
reducing the complexity and they're
going for what and I can't answer that I
don't know when the payoff is going to
be there's uncertainty there I don't
know how deep the rabbit-hole goes
and also I don't know what the payoff
will be if I could put if I could
quantify either of those I might be in
with a shout but I'm not so I'm kind of
going on instinct I really really think
we should clear up something here
because otherwise the will is going to
fall off shut up and ship it okay we'll
do that then so side effects here what
the desperately sad side effect here is
that we lose key skills we lose the
ability to be frustrated with that code
and to use their energy direct their
energy - simplifying things things as
simple as wrapping big bunches of stuff
in some kind of interface some kind of
barrier or I can say everything behind
there is a bit weird but basically what
it does is pry straight no idea how but
I know if I put enough trade data and
market data and I get a number back and
I can make decisions off that number and
so now I can start reasoning about that
things like looking for seams in big
code bases we lose the skills to do that
okay because we never do it so okay
here's the bad news those painkillers
are everywhere I want to give you a few
examples and hopefully get you feeling
as uncomfortable as ice
about some of the stuff that we have and
also I'm hoping you're going to look at
this and go wait but that's just a thing
we do because that's why it's subtle
right one of the challenges is but
that's the way we've always done it I
don't buy but that's the way we've
always done it um so yeah let's see
here's a few examples we look
architecture we look at design we're
going look at tooling some techniques
programming techniques we look at
process organization just a few quick
examples okay I just need to get a sense
of where the painkillers are so
enterprise mandates we will use such and
such a design stack who's who was who
lived through the joys of everything has
to be son blueprint j2ee I want I want
JSP is talking to servlet talking to
session beings talking to entity beans
talking to a dao talking to a database
to change a field on a webpage that so
now I want to I want to change the name
of the field heaven forbid right Oh
crikey or a thing that was an INT now
needs to become a float okay so we play
get set ping pong get that set get
that's in that get that's it that's it
let go back to that insert doesn't work
set like an active oh I don't know
deploy boom okay Enterprise manage I'm
working with a company at the moment who
recently decided that everything was
going to be in Python everything all new
software's gonna be in Python okay
there's a wonderful quote from a comedy
show in the UK from the 80s those that
they were running an advert on TV for a
quite a long time and it was trying to
get people to move to a new town in the
UK called Milton Keynes and it said
wouldn't it be lovely if all cities were
like Milton Keynes and it would show you
these are Dilek shots of Milton Keynes
this town town of Milton Keynes but
anyways this beautiful and and and so
the response of his comedy show is it
would be dreadful if all cities were
like any city
wouldn't that be dreadful
yeah we don't all seems to be like any
city and this is the same thing wouldn't
it be dreadful if all systems are like
any system right because surely that's
not about fit for purpose that's about
some blanket mandate and so we get a lot
of this we get a lot of this blanket
enterprise mandates and again it's for
our own good it's because most of us are
dumb right we shouldn't be allowed out
anyway certainly should be coding on our
own pair programming is good cuz kinda
like parenting it's kinda you've got
you've got a buddy right hands like if
you haven't got a buddy okay go pay with
that person so enterprise mandates so
what else then design undirected local
choices it took me a long time to come
up with this phrase I'm fine with local
choices I love people making I love
federated decision making okay does
anybody know what federated means I will
give me the ref definition federated
it's the decision is allowed to happen
locally within a strictly defined
decision-making framework okay so you
have local law is still made under a law
making framework it's still executed
under the same way as any other local
law there are guidelines it's not all go
nuts okay so we are Utah and we're going
to go with anarchy okay okay that's not
going to happen in Utah because everyone
there is far too nice okay we could go
to Texas everyone let's get our word
that kind of what he isn't it so so the
point is this is undirected and one of
the things I think that's happened as a
dreadful shame in the last 10 years 15
years with the kind of onslaught the
advent of all these agile methods is we
talked about we talked about kind of
empowering teams and we say want you
guys to be self-organizing okay I'm fine
with self-organizing now there's another
term though that sits alongside of
self-organizing which is self directing
self directing says we will decide what
we're going to do self-organizing says
we will figure out how we're going to
get there so General Eisenhower one of
us all great 20th century generals he
used to get these insanely
like he used to get the kind of results
from his men that no other generals
would get and they said how do you do
that it is really easy I don't tell them
what to do I tell them what I need to
happen but I don't tell them what to do
is said if you don't tell people what to
do you will be amazed at how ingenious
and creative they can be so I don't tell
them I tell them the objective where
we're doing this sort of agile hands-off
stuff particularly at scale in
organizations is we're saying oh
management is bad telling people what to
do is bad ergo none of that let's just
see what happens okay and what happens
it turns out is a mess in most cases so
what we actually want to do is we want
to have direction on a set direction
without being directive we want to say
this is where we're going without
micromanaging the hell and then you'll
be amazed again at how well people will
do the hell that's awesome
without giving them a why or what for
they'll just do stuff and if you're
lucky enough that that stuff coincides
with anything you care about well done
okay but it's pretty unlikely
so undirected local choices don't roll
up you get you get chaos local choices
languages or a local choice okay so in
the UK we have English we have Welsh we
have Gaelic in Scott in Ireland these
are all local choices they may merge
locally and locally they're fine when
people meet at the edges they mostly
fight okay they certainly don't
understand each other until they all
speak English okay English happen to win
so in fact French and Latin one for a
while but English ended up winning and
there's no kind of gentle transition
from Gaelic to English you just learn
English
there's no gentle transition from Welsh
to English they're different languages
so these local decisions made sense
locally they don't roll up you will not
get that rolling up what else i des Eaux
your ide is a tool for masking
complexity okay so at a fantastic blog
post and the comments on it will
wonderfully by Lee as well like how dare
you don't understand programming you
fool and it's
I said basically said your IDE is is a
facade over how crap your language is so
this look at something like Java okay
Java as a language it's okay I mean I've
been using Java for many many years I've
kind of gotten used to its warts and
stuff here's the thing Java has a very
early language decision said we will
couple class names to file names they
will be the same thing so if you change
a class name in Java given if I change a
class name in Python I do some typing
I'm done if I change a class name in
Java say I've got a sheep dot Java I
know a class called sheep and I want to
turn it into a class called cheese for
some reason then I don't just replace
the word sheep with the word cheese I
create a new file called cheese Java
copy sheep Java mostly except I'm
replacing some of the words on the way
through delete the original file and
then go and look for any references to
it anywhere else in the codebase and
change those atomically the fun starts
when I start changing package names
because of package name references of
directory so now I've got an arbitrarily
deep nested file structure underneath
some point and what I'm going to do is
create a sibling file structure next to
it that replicates it all copy all the
files in mostly because I'll change the
package name remember and put all those
in and then delete recursively
depth-first all of the old files and
then go through every reference to those
to the package name and change all that
atomically right if people had to use
regular text editors to use Java no one
would be using Java it's just too much
of a pain or at least all be refactoring
in Java yeah and what that might have
caused is that might have caused enough
back pressure early in Chavez life that
they rescinded such a ridiculous
decision but they didn't because IDs
okay and so again these took that it's a
fantastic tool mostly what it's doing is
masking the dumbness of some of the core
language decisions so most people I know
who are programming in Ruby or Python or
or JavaScript will mostly use vim Emacs
no not Emacs but like you know TextEdit
or sublime text but I don't know anyone
closure people you have to be smart
enough to understand closure to be
masochistic enough to enjoy Emacs I
don't know at some relationship it build
automation I used to be a huge huge
advocate of automating bills
oh no what's he going to say about
builds I need to speed up is I going to
say so build automation it sounds like a
sensible thing to do automate your build
make it repeatable no your build becomes
the rug under which you sweep every
crappy infrastructure decision you make
all the way through your product until
you end up with on the on one of the
largest projects I worked on that 10
years ago now there was a build.xml
sorry there was a build script that
would take a bunch of candidate XML
files put them through an XSLT generate
a bunch of build files which would
generate build files which ended up
being about two Meg's of build rules or
a Java app at the time it made sense
every single one of those tiny decisions
made sense yeah oh my word so I'm not
saying these things are bad I'm saying
they're painkillers as well as whatever
else they might be okay so revisit these
things TDD I don't need to talk about CV
because apparently it's dead okay not
bad it's not dead
the thing with TDD I do a lot less TDD
than I used to I don't do none I'm a
huge fan of TVD the thing is TDD III
think of TDD is deliberate ignorance I
am going to in order to reason about
this piece of code I am going to have to
isolate it from all the other really
ugly messy bits of code because they
don't make sense or rather I can't fit
them in my head I can't fit the whole
system in my head therefore in order to
test this thing I need to reason about
it in the small and so I write these
very small tests it's fantastic for
doing small scaled targeted design it's
a brilliant brilliant design method
within the construct of a larger
strategy okay there's a second half to
that sentence right so again I'm not
saying don't do TDD I'm not saying don't
automate your build I'm saying be aware
that as well ever
it's giving you it's also acting as a as
a painkiller to the complexity around G
process too much process too little
something like scrum back in the 90s
when it was invented it was exactly the
right amount of process it would take
months and months and months to do
anything scrum came along and they said
actually we take years to do something
scrum came along and said we can do this
in months and and prove the impossible
the same process 20 years later is now a
little bit silly right it's now we are
going to get together every two weeks
right if you're listening to Adrienne
Cockcroft this morning he can build
deploy test destroy and abandon half a
dozen products in two weeks
why on earth would you want two weeks to
be your planning schedule that's just
bonkers two hours maybe two days at a
stretch two weeks I was sorry I've
changed companies twice right what
weight are you talking about we've we've
pivoted we've pivoted there's no longer
even the thing anymore and and finally
at one dimension Conway's law so
organization when you have
organizational complexity we we buffer
ourselves by creating software
complexity if the upstream and
downstream guys are unreliable or
unresponsive or I can't trust them
then what I do is I I put that distrust
into my code and this is why
architecture at scale starts to carry
the organization or the communication
structures of the organization you're in
which is Conway's law is because we're
trying to get work done and the way we
get work then is we insulate ourselves
the way to really get work done is to go
upstream and speak to those guys and
engage them or to go downstream and
speak to those guys and engage them when
we're developers going upstream is about
creating collaboration with stakeholders
right when you're going downstream
that's about creating collaboration with
operations folks and DevOps and all that
good stuff so these are things these are
things that we can do we don't do that
we surround ourselves with interfaces
that'll show so okay I have 12 minutes
left and just on to the second section
so that's okay don't know it sounds a
reducing complexity then
that hard the hardest bit about reducing
complexity is recognizing complexity
okay monitoring your own cognitive loads
a cognitive research cognitive
psychology suggests that there's a thing
called cognitive load cognitive load is
how hard it is to reason about a thing
so monitoring a cognitive load means
this means is the thing I'm reasoning
about should it be this hard to reason
lovely illustration he says he says
there are some problems I love wrestling
with I come out of that problem smarter
I come out of that problem having
developed you know as a programmer or as
a product builder or something I come
out
he said connecting to a printer isn't
one of those right connecting to a
printer is our crap here we go again got
to connect to a printer you're kidding
three hours later still trying to
connect to a printer right brilliant you
know that's so so cognitive load connect
to a printer three hours really hard
work for a very very smart programmer
that's not Matt that's not that doesn't
match okay so if you start to spot a
cognitive load disproportionate
indicator that there's probably
something you can do okay so the next
thing to try and develop is to see
what's really there so terry pratchet
writes about these witches in this way
this is this world books and one of
these witches she has the gift of first
sight I said don't you mean second sight
he says no she sees what's really there
oh yeah so seeing what's really there
this is this is surprisingly difficult
so Joe warms her bunchy folks know so he
was I was working with him and we were
writing lots of Java stuff and it was
quite webby and so we were kind of using
Java web stuff and he said that servlet
thing who uses that like oh forgot that
was in there it's just kind of part of
the world right it's just it we've just
always had that
we've always had in Java where we've
always had the servlet container and the
servlet container has life cycle has
filters interceptors tear up tear down
them of a bunch of wait so Joe went off
and re-implemented he wrote a HTTP
server in Java because there aren't
enough of those but it turns out his
just does that his is just an HTTP
server no anything else it has as a
consumer it has one interface the one
interface is called a request Handler it
has one method on it guess what the
method is handle request that's it kind
of easy to test kind of easy to
understand start in about a millisecond
okay it's not a bad stack yeah why on
earth would someone sit there and write
a new web server because he would he saw
what's really there he has the gift of
seeing what's really there this is the
big one if you can create consistency
you can create shared idioms shared
principles then you start getting a huge
win off the back of that the first thing
is this is we don't want everything to
be identical I don't want all of you
guys to be doing the same thing that's
silly
but if we are working off the same set
of shared principles principles plus
some context you apply them that
determines your practices in this
context based on these share principles
I will do this thing it makes sense to
do this thing and what it means is if
someone else were to come along with the
same guiding principles they're likely
to make the same decision which reduces
cognitive load I can look in a situation
and I can say ah you've done this thing
you've made this decision I can see why
you did that
because it's kind of similar to what I
would have done alright
difference is data this is a meme I've
been trying to get out there as as soon
as you have a certain degree of
consistency I can now start seeing
inconsistency as signal if we have say a
we all decide within this particular
architecture this particular system is
particular whatever that mostly when
we're sending messages around will
probably do it overrated
he will probably use Jason it's an
arbitrary decision we could send it over
a message bus and use AMQP I don't care
whatever decision we make we all decide
we're going to make it except that I see
Matt written something and it's using
direct memory allocation between two
processes talking some binary protocol
what does that tell me that tells you
one of two things or one of three things
it tells me Matt didn't get the memo
right or it tells me that's all that
matters whether or not he got the memo
or what it might tell me is Matt got the
memo Thanks
this is a special case this might be a
low latency fast path case and that's
why he's on it in any case I look at
that and it because it's not like the
other kids that signal and I go up
there's something interesting over there
if instead every single messaging
decision that was local completely
undirected and totally arbitrary it
would all be noise every single bit of
messaging going anywhere would just be
arbitrary okay and so then I wouldn't
know that Matt had made that decision
for that reason he just made that
decision because he read something on
hacker news that morning you know that
was cool obviously that wouldn't happen
because it's hacker news but okay so and
and I guess the biggest takeaway that I
had for um for trying to create
consistency is is the law is when I
realize familiarity and simplicity of
different things you get used to
crappiness
what's happening is the crappiness is
bending your brain in such a way that
it's now okay with the crap in it you're
kind of creaking into a shape like that
in you're going oh that's not so bad
yeah I could do that okay right you're
broke that's happening in your brain and
then someone comes along and says how
does that work he got out you just got
to be this shape and it'll all make
sense okay and and so we've become
familiar with a thing but that doesn't
make the thing simple and so one of the
things I try and do is whenever I can
I'll get some fresh eyes into a
situation and I'll say what's the
dumbest thing you see because if you say
something like they get someone new in
the team and they say
is there anything you'd like to change
they're all on their best behavior
because they new again no no no so fine
it's like what's the dumbest thing you
say and they say well your bill takes
ten minutes which seems a bit dumb given
the speed you try to move that and the
and then suddenly go hmm he criticized
the build oh no right let me tell you
six months to get the build down from
the river that's a ten minute I'll have
you know we've worked long and hard
still ten minutes still ten minutes
it still beyond the adrian Cockcroft
shiny item threshold of like you know
ten seconds or whatever I'm still not
okay and he's right or she's right
they're right they come and they say
that build it takes too long because I
know the history of the build and I've
got the baggage with that build I'm fine
with ten minutes I'm actually not fine
into shape to be okay with it
okay new person is like no that's a
weird thing that's not okay
so try and get that in a quick Tale of
Two Cities this is doesn't come terribly
well hit this is central London here
we've got Bank to some polls and through
to Tottenham Court Road
this is Chicago this road here Americans
love this this is cheap side then it
becomes Holborn then it becomes High
Holborn and then it becomes shall spree
Avenue see that that's all one road they
love it when we do that we keep changing
the road numbers the side of the road
and the reason that happened is London
is really lots and lots and lots of
little villages that blurred at the
edges chicago bit planned this one here
is I know can you can even see that
which one is it Randolph I think it's
Randall Randall is bed straight and it
goes along there and it goes along then
it goes on it goes on there too about
infinity I think it ends up at like you
know Japan or somewhere right
State Street State Street goes north
carries on down till you get to yeah you
get the idea
incredibly long straight very simple and
and they've even got the numbering so
you number it by by by block and you can
how far away it is we don't do that we
put 28
to 162 for giggles this will Fox the
Americans yeah now cause any any
literary any any literary people among
you will know that Tale of Two Cities
was nothing to do with America it was
about the two cities were London and
Paris Paris is really interesting city
this looks a bit of a mess right until I
tell you it's designed on a spiral so
that's the first it's a Ronda's ones
which are like boroughs one two three
four five six seven eight nine ten
eleven twelve all the way around 220
okay do it again
it's called less scale go the snail okay
so they designed their so what happened
was us in the nineteenth century they
they realized I already had an existing
city they couldn't just raze it to the
ground and grid it up okay so what they
did is they took the area's they had and
they owned them in a spiral because this
happens I can be if I'm anywhere here
every single street corner has their on
this Mont on it as well as a street name
I know I'm in the ninth if I walk for a
few hundred yards in any direction it
will no longer say nine it will start
saying two or eight or ten or 18 and as
soon as that happens I know exactly
where I am and which direction I'm
traveling how cool is that you don't
need a grid they've hacked around that
so this is quite a nice nice model
anyway meaning on avoiding complex in
the last thing I need to get to avoiding
complexity I guess the first thing
really is to be aware to be comfortable
with the fact that complexity is always
the default state what do I mean by that
I mean there isn't a physics so in our
universe we have physics so if I look at
activity here and I look at activity
here it's governed by similar laws okay
and then if I start seeing activity over
here that's weird like water climbing up
the side of a glass or electrons
spinning in the wrong direction I can be
fairly sure that something local here is
very very unusual
okay so people's subjective behavior
doesn't have physics it doesn't have
governing laws governing rules which
means that
fairly arbitrary things are going to
happen locally even sensible sounding
things Gaelic makes a lot of sense to
Gaelic speakers Welsh makes a lot of
sense to Welsh speakers the way that
language evolved made a lot of sense
okay
the Welsh probably have twenty eight
words for sheep and probably 200 words
for rain okay but the English probably
have fifty words for fight right it's
just kind of video it's just it's
contextual so so because there's just
local decisions if we want anything to
roll up and we need to create the
physics we need to define the rules we
need to all agree what the rules are and
then we can make local decisions that
will roll up what else we say it grows
one day at a time there's a lovely Fred
Brooks quote from the mythical man-month
where he's looking at a project and the
project is now running a year late and
he's standing in front of the US Senate
or something and saying well you know
professor Brooks how on earth how did
the project get to be a year late it's a
one day at a time
that's how project gets to be a year
late okay
complexity grows one little decision at
a time that's it okay a consistency at
scale that means is a daily choice every
single day there will be forces the way
work works the way organizations work
there will be things that are trying to
cause us to to create inconsistency
inconsistency is like entropy it's it's
the the thermodynamics of knowledge if
you like okay it's going to get messy
keeping it not messy is a daily choice I
know so again we need to agree idioms
and guiding principles once we agree
those idioms and guiding principles then
what else we're going to say so and then
we're going to strive for simplicity we
can strive for it it can be a thing we
going to make it easier certain certain
tools certain languages certain tool
chains have affordances that make it
more likely that we'll have consistency
certain ones don't I was going to
mention Scala and closure as a couple of
examples I think scholars of a great
language it's just very very uh
opinionated I could do anything in it
Clojure has parens that's most of
closure right there
okay so because it's so simple they went
this is this is my programming too easy
we should learn Emacs that's that's
that's how that went so let me just
quickly sum up before I get rushed off
the stage complexity is the default
condition okay
absent any decisions we make absent
anything else complexity will grow
entropy will increase okay
local decisions will get made will
appear to make sense locally will not
roll up will clash at the edges we
always we always have the choice to
simplify we only have the choice to
complexity is and if we can arm
ourselves with techniques that are going
to help us do that and consistency is
the key consistency is the mechanism by
which we can make we can reduce
cognitive load we can or make cognitive
load appropriate some problems are
intrinsically hard that's fine they
don't need to be any harder than they
are it really shouldn't be this
drive time for any questions a couple
math questions anyone one s fun we have
an infinite amount of time for questions
at this pace
there we go question sir Oh especially
front so when you're aiming for
consistency if you come into a
particular situation that you could do
one of two ways maybe it's a coding
style something like that and you don't
know which is better yep
when do you choose one and set a
standard that's a really good question
so all right so we've got a choice to
make
okay and there's the the lovely Yogi
Berra quote is when you when you come to
a crossroads take it okay very
comforting fork in the road take it the
problem with that is that that's that's
rooted in the physical world so when I
come across a situation where you know
okay we're on the JVM we've been using
Java for a while we could go with Scylla
we could go with cloture we could stick
with Java we can look at some other jvm
languages what should we do my answer is
always yes right yes we could do any of
those four things let's try those four
things let's try those four things in
parallel let's try those four things in
okay it's called set based engineering
so the idea is we're going to do all of
the things and at some point data will
emerge at some point it will become
obvious that one of these is a better
fit for the kind of things we're doing
than the others and I suspect it won't
be the one we thought of so I had I had
a real example of this about a year ago
I was doing my client on a class called
accelerated agile and I was doing this
class and two successive classes I had
the same question
we're on JVM we're using Java closure or
Scala down what do you think cloud or
scarlet and I said yes I said yes
closure or Scala or maybe Java right so
the teams in both cases went off there
about six people in the team so they had
a different pairs spent a couple of
weeks one one pedaling Scala one day
Clojure one known Joe Java for a couple
of weeks and then they all swap round
one so by the end of four weeks they all
had experience of two of the candidate
languages in the same problem space and
after that they said well what do we
think as a team and they all went that
one now the great thing about
set base engineering it's inefficient
but it's very effective so it costs
three times as much to build this thing
make you already started doing it four
weeks ago yeah so you haven't lost any
time you've just lost a bit of bandwidth
and so so this is how Boeing designs
wings for instance and nowadays I think
it's law actually for a lot of safety
critical systems like if you're building
a braking software in cars and anti-lock
braking you will get three different
companies to build this same same code
same system you pay all of them it's not
like a cost of sale thing you pay all of
them and then you use each two as the
testbed for the third one and then you
end up going with whichever one you like
and so again that means it costs three
times as much but you end up with a
system that you are now absolutely
confident is reasonably stable and you
had a choice of three two to choose from
so Boeing does it with wings cars car
systems do it with safety-critical
fly-by-wire systems all that kind of
stuff and then you throw the other two
away so you're not left with
inconsistency yeah exactly
all what you may do is you may say we're
going to go with this one we're going to
go with Scala but we noticed a really
nice bunch of programming paradigms that
we picked up enclosure that's going to
inform our programming style in scarlet
we quite like that it's a gentle path
from Java we also we're not happy with
the fact there's 15 different ways to do
looping so we're going to use recursion
because we quite like recursion we're
going to use map and reduce we're not
okay well that's that's an idiom that's
a design choice we've made based on our
experience in some other languages and
so it's not just it's not binary one two
could be we're going to take this but
we're going to inform it with these
other things Understanding between
people at lunch that's always a bad
place to be
we'll do one more question and then
lunch I'm around for the whole thing say
okay the question when you're sort of
trying to reduce complexity when you're
designing something about detecting
things and generally designing for the
general case but then you get situations
where things are too general and don't
really solve any problem well overseas
over architecting something and
something was very specific but it's
very fragile how do you sort of keep
things simple but avoid one or the other
so there's a chap called Chris Matz who
talks a lot about real options and he
has a lovely say he looks at the phrase
YAG me and he says you ain't going to
need it so the point of real options or
thinking in options is that you should
you should make decisions deliberately
in other words you should defer a
decision okay for as long as makes sense
to defer it you shouldn't commit early
unless you've got a good reason okay if
you've got a good reason commit early
for sure if you don't have a good reason
don't commit early unless you know why
so yagna is only the first half of that
sentence just says don't commit early
right it says are you're not going to
need it okay so don't commit early
unless you know why it is you might
going to need it yeah so for instance if
I'm building something and I know fairly
soon there's going to be not just a
single case but two okay there's going
to be more than one then there's nothing
wrong with me designing for the more
than one case now I'm not going to make
an uber generic thing I'm gonna be aware
that this thing is going to need to
support is likely to need to support
more than one cases fairly soon so I
don't need to code them all in now what
I probably going to do while I'm
thinking about it is leave the seams
there make it obvious when someone comes
back and the obvious miss thing is
really important make it obvious when
someone comes back where they need to
make that change oh look he left the
hook in here that I could do this thing
it's a very small hook if no one ever
needs it it's not going to get in the
way but it just makes it easier I'm sort
of thinking ahead because the person who
I might be handing this off to might be
me right so it's slightly selfish but
the there's also something I've been
talking about for a long time and
is this idea of a change event horizon
so a change event horizon is how long
after I make a dumb decision how soon
recant that dumb decision can i reverse
that decision when I'm writing in my
word I'll write business software I
write enterprise software typically my
change event horizon is about a week
right in an emergency it can be hours
I just stuff this thing up within hours
designing an API a programming language
a hugely used library something like
that right
your change event horizon if you put
something dumb into c-sharp your change
event horizon might be 30 years yeah I
mean literally there are some dumb
decisions in Java that are still there
okay people are still using new Java
util vector okay
vector was deprecated in Java 1.1 that's
before a lot of you were born right so
people still use Java util vector and so
they went okay look we can't get rid of
vector what we'll do is well I'll tilt
will do will get will get underneath
vector and we actually make it a list
people they need never know right whew
there I fixed it bit of gaffer tape
under it yeah so if you have a long
change event horizon then it means that
you need to be much more careful about
the decisions you make so it's again
it's a deliberateness thing just be
aware of the context you're in and let
that inform how far out you might those
decisions okay I think that's it