Bob Jekyll was already sitting at a table, sipping a pint of Black Sheep and nibbling on a bowl of peanuts when Hugh and Louise arrived.
<Hugh> Hello, are you Bob?
<Bob> Yes, indeed! You must be Hugh and Louise. Can I get you a thirst quencher?
<Louise> Lime and soda for me please.
<Hugh> I’ll have the same as you, a Black Sheep.
<Bob> On the way.
<Hugh> Hello Louise, I’m Hugh Lewis. I am the ops manager for acute medicine at St. Elsewhere’s Hospital. It is good to meet you at last. I have seen your name on emails and performance reports.
<Louise> Good to meet you too Hugh. I am senior data analyst for St. Elsewhere’s and I think we may have met before, but I’m not sure when. Do you know what this is about? Your invitation was a bit mysterious.
<Hugh> Yes. Sorry about that. I was chatting to a friend of mine at the golf club last week, Dr Bill Hyde who is one of our local GPs. As you might expect, we got to talking about the chronic pressure we are all under in both primary and secondary care. He said he has recently crossed paths with an old chum of his from university days who he’d had a very interesting conversation with in this very pub, and he recommended I email him. So I did. And that led to a phone conversation with Bob Jekyll. I have to say he asked some very interesting questions that left me feeling a mixture of curiosity and discomfort. After we talked Bob suggested that we meet for a longer chat and that I invite my senior data analyst along. So here we are.
<Louise> I have to say my curiosity was pricked by your invitation, specifically the phrase ‘system behaviour charts’. That is a new one on me and I have been working in the NHS for some time now. It is too many years to mention since I started as junior data analyst, fresh from university!
<Hugh> That is the term Bob used, and I confess it was new to me too.
<Bob> Here we are, Black Sheep, lime soda and more peanuts. Thank you both for coming, so shall we talk about the niggle that Hugh raised when we spoke on the phone?
<Hugh> Ah! Louise, please accept my apologies in advance. I think Bob might be referring to when I said that “90% of the performance reports don’t make any sense to me“.
<Louise> There is no need to apologise Hugh. I am actually reassured that you said that. They don’t make any sense to me either! We only produce them that way because that is what we are asked for. My original degree was geography and I discovered that I loved data analysis! My grandfather was a doctor so I guess that’s how I ended up in doing health care data analysis. But I must confess, some days I do not feel like I am adding much value.
<Hugh> Really? I believe we are in heated agreement! Some days I feel the same way. Is that why you invited us both Bob?
<Bob> Yes. It was some of the things that Hugh said when we talked on the phone. They rang some warning bells for me because, in my line of work, I have seen many people fall into a whole minefield of data analysis traps that leave them feeling confused and frustrated.
<Louise> What exactly is your line of work, Bob?
<Bob> I am a systems engineer. I design, build, verify, integrate, implement and validate systems. Fit-for-purpose systems.
<Louise> In health care?
<Bob> Not until last week when I bumped into Bill Hyde, my old chum from university. But so far the health care system looks just like all the other ones I have worked in, so I suspect some of the lessons from other systems are transferable.
<Hugh> That sounds interesting. Can you give us an example?
<Bob> OK. Hugh, in our first conversation, you often used the words “demand” and “capacity”. What do you mean by those terms?
<Hugh> Well, demand is what comes through the door, the flow of requests, the workload we are expected to manage. And capacity is the resources that we have to deliver the work and to meet our performance targets. Capacity is the staff, the skills, the equipment, the chairs, and the beds. The stuff that costs money to provide. As a manager, I am required to stay in-budget and that consumes a big part of my day!
<Bob> OK. Speaking as an engineer I would like to know the units of measurement of “demand” and “capacity”?
<Hugh> Oh! Um. Let me think. Er. I have never been asked that question before. Help me out here Louise. I told you Bob asks tricky questions!
<Louise> I think I see what Bob is getting at. We use these terms frequently but rather loosely. On reflection they are not precisely defined, especially “capacity”. There are different sorts of capacity all of which will be measured in different ways so have different units. No wonder we spend so much time discussing and debating the question of if we have enough capacity to meet the demand. We are probably all assuming different things. Beds cannot be equated to staff, but too often we just seem to lump everything together when we talk about “capacity”. So by doing that what we are really asking is “do we have enough cash in the budget to pay for the stuff we thing we need?”. And if we are failing one target or another we just assume that the answer is “No” and we shout for “more cash”.
<Bob> Exactly my point. And this was one of the warning bells. Lack of clarity on these fundamental definitions opens up a minefield of other traps like the “Flaw of Averages” and “Time equals Money“. And if we are making those errors then they will, unwittingly, become incorporated into our data analysis.
<Louise> But we use averages all the time! What is wrong with an average?
<Bob> I can sense you are feeling a bit defensive Louise. There is no need to. An average is perfectly OK and is very useful tool. The “flaw” is when it is used inappropriately. Have you heard of Little’s Law?
<Louise> No. What’s that?
<Bob> It is the mathematically proven relationship between flow, work-in-progress and lead time. It is a fundamental law of flow physics and it uses averages. So averages are OK.
<Hugh> So what is the “Flaw of Averages”?
<Bob> It is easier to demonstrate it than to describe it. Let us play a game. I have some dice and we have a big bowl of peanuts. Let us simulate a simple two step process. Hugh you are Step One and Louise you are Step Two. I will be the the source of demand.
I will throw a dice and count that many peanuts out of the bowl and pass them to Hugh. Hugh, you then throw the dice and move that many peanuts from your heap to Louise, then Louise throws the dice and moves that many from her pile to the final heap which we will call activity.
<Hugh> Sounds easy enough. If we all use the same dice then the average flow through each step will be the same so after say ten rounds we should have, um …
<Louise> … thirty five peanuts in the activity heap. On average.
<Bob> OK. That’s the theory, let’s see what happens in reality. And no eating the nuts-in-progress please.
They play the game and after a few minutes they have completed the ten rounds.
<Hugh> That’s odd. There are only 30 nuts in the activity heap and we expected 35. Nobody nibbled any nuts so its just chance I suppose. Lets play again. It should average out.
….. …..
<Louise> Thirty four this time which is better, but is still below the predicted average. That could still be a chance effect though. Let us run the ‘nutty’ game this a few more times.
….. …..
<Hugh> We have run the same game six times with the same nuts and the same dice and we delivered activities of 30, 34, 30, 24, 23 and 31 and there are usually nuts stuck in the process at the end of each game, so it is not due to a lack of demand. We are consistently under-performing compared with our theoretical prediction. That is weird. My head says we were just unlucky but I have a niggling doubt that there is more to it.
<Louise> Is this the Flaw of Averages?
<Bob> Yes, it is one of them. If we set our average future flow-capacity to the average historical demand and there is any variation anywhere in the process then we will see this effect.
<Hugh> H’mmm. But we do this all the time because we assume that the variation will average out over time. Intuitively it must average out over time. What would happen if we kept going for more cycles?
<Bob> That is a very good question. And your intuition is correct. It does average out eventually but there is a catch.
<Hugh> What is the catch?
<Bob> The number of peanuts in the process and the time it takes for one peanut to get through is very variable.
<Louise> Is there any pattern to the variation? Is it predictable?
<Bob> Another excellent question. Yes, there is a pattern. It is called “chaos”. Predictable chaos if you like.
<Hugh> So is that the reason you said on the phone that we should present our metrics as time-series charts?
<Bob> Yes, one of them. The appearance of chaotic system behaviour is very characteristic on a time-series chart.
<Louise> And if we see the chaos pattern on our charts then we could conclude that we have made the Flaw of Averages error?
<Bob> That would be a reasonable hypothesis.
<Hugh> I think I understand the reason you invited us to a face-to-face demonstration. It would not have worked if you had just described it. You have to experience it because it feels so counter-intuitive. And this is starting to feel horribly familiar; perpetual chaos about sums up my working week!
<Louise> You also mentioned something you referred to as the “time equals money” trap. Is that somehow linked to this?
<Bob> Yes. We often equate time and money but they do not behave the same way. If have five pounds today and I only spend four pounds then I can save the remaining one pound for tomorrow and spend it then – so the Law of Averages works. But if I have five minutes today and I only use four minutes then the other minute cannot be saved and used tomorrow, it is lost forever. That is why the Law of Averages does not work for time.
<Hugh> But that means if we set our budgets based on the average demand and the cost of people’s time then not only will we have queues, delays and chaos, we will also consistently overspend the budget too. This is sounding more and more familiar by the minute! This is nuts, if you will excuse the pun.
<Louise> So what is the solution? I hope you would not have invited us here if there was no solution.
<Bob> Part of the solution is to develop our knowledge of system behaviour and how we need to present it in a visual format. With that we develop a deeper understanding of what the system behaviour charts are saying to us. With that we can develop our ability to make wiser decisions that will lead to effective actions which will eliminate the queues, delays, chaos and cost-pressures.
<Hugh> This is possible?
<Bob> Yes. It is called systems engineering. That’s what I do.
<Louise> When do we start?
<Bob> We have started.