[Bzzzzz Bzzzzz] Bob’s phone was on silent but the desktop amplified the vibration and heralded the arrival of Leslie’s weekly ISP coaching call.
<Bob> Hi Leslie. How are you today and what would you like to talk about?
<Leslie> Hi Bob. I am well and I have an old chestnut to roast today … target-driven-behaviour!
<Bob> Excellent. That is one of my favorite topics. Is there a specific context?
<Leslie> Yes. The usual desperate directive from on-high exhorting everyone to “work harder to hit the target” and usually accompanied by a RAG table of percentages that show just who is failing and how badly they are doing.
<Bob> OK. Red RAGs irritating the Bulls eh? Percentages eh? Have we talked about Ratio Hazards?
<Leslie> We have talked about DRATs … Delusional Ratios and Arbitrary Targets as you call them. Is that the same thing?
<Bob> Sort of. What happened when you tried to explain DRATs to those who are reacting to these ‘desperate directives’?
<Leslie> The usual reply is ‘Yes, but that is how we are required to report our performance to our Commissioners and Regulatory Bodies.’
<Bob> And are the key performance indicators that are reported upwards and outwards also being used to manage downwards and inwards? If so, then that is poor design and is very likely to be contributing to the chaos.
<Leslie> Can you explain that a bit more? It feels like a very fundamental point you have just made.
<Bob> OK. To do that let us work through the process by which the raw data from your system is converted into the externally reported KPI. Choose any one of your KPIs
<Leslie> Easy! The 4-hour A&E target performance.
<Bob> What is the raw data that goes in to that?
<Leslie> The percentage of patients who breach 4-hours per day.
<Bob> And where does that ratio come from?
<Leslie> Oh! I see what you mean. That comes from a count of the number of patients who are in A&E for more than 4 hours divided by a count of the number of patients who attended.
<Bob> And where do those counts come come from?
<Leslie> We calculate the time the patient is in A&E and use the 4-hour target to label them as breaches or not.
<Bob> And what data goes into the calculation of that time?
<Leslie>The arrival and departure times for each patient. The arrive and depart events.
<Bob>OK. Is that the raw data?
<Leslie>Yes. Everything follows from that.
<Bob> Good. Each of these two events is a time – which is a continuous metric. In principle, we could in record it to any degree of precision we like – milliseconds if we had a good enough enough clock.
<Leslie> Yes. We record it to an accuracy of of seconds – it is when the patient is ‘clicked through’ on the computer.
<Bob> Careful Leslie, do not confuse precision with accuracy. We need both.
<Leslie> Oops! Yes I remember we had that conversation before.
<Bob> And how often is the A&E 4-hour target KPI reported externally?
<Leslie> Quarterly. We either succeed or fail each quarter of the financial year.
<Bob> That is a binary metric. An “OK or not OK”. No gray zone.
<Leslie> Yes. It is rather blunt but that is how we are contractually obliged to report our performance.
<Bob> OK. And how many patients per day on average come to A&E?
<Leslie> About 200 per day.
<Bob> So the data analysis process is boiling down about 36,000 pieces of continuous data into one Yes-or-No bit of binary data.
<Leslie> Yes.
<Bob> And then that one bit is used to drive the action of the Board: if it is ‘OK last quarter’ then there is no ‘desperate directive’ and if it is a ‘Not OK last quarter’ then there is.
<Leslie> Yes.
<Bob> So you are throwing away 99.9999% of your data and wondering why what is left is not offering much insight in what to do.
<Leslie>Um, I guess so … when you say it like that. But how does that relate to your phrase ‘Ratio Hazards’?
<Bob> A ratio is just one of the many ways that we throw away information. A ratio requires two numbers to calculate it; and it gives one number as an output so we are throwing half our information away. And this is an irreversible act. Two specific numbers will give one ratio; but that ratio can be created by an infinite number possible pairs of numbers and we have no way of knowing from the ratio what specific pair was used to create it.
<Leslie> So a ratio is an exercise in obfuscation!
<Bob> Well put! And there is an even more data-wasteful behaviour that we indulge in. We aggregate.
<Leslie> By that do you mean we summarise a whole set of numbers with an average?
<Bob> Yes. When we average we throw most of the data away and when we average over time then we abandon our ability to react in a timely way.
<Leslie>The Flaw of Averages!
<Bob> Yes. One of them. There are many.
<Leslie>No wonder it feels like we are flying blind and out of control!
<Bob> There is more. There is an even worse data-wasteful behaviour. We threshold.
<Leslie>Is that when we use a target to decide if the lead time is OK or Not OK.
<Bob> Yes. And using an arbitrary target makes it even worse.
<Leslie> Ah ha! I see what you are getting at. The raw event data that we painstakingly collect is a treasure trove of information and potential insight that we could use to help us diagnose, design and deliver a better service. But we throw all but one single solitary binary digit when we put it through the DRAT Processor.
<Bob> Yup.
<Leslie> So why could we not do both? Why could we not use use the raw data for ourselves and the DRAT processed data for external reporting.
<Bob> We could. So what is stopping us doing just that?
<Leslie> We do not know how to effectively and efficiently interpret the vast ocean of raw data.
<Bob> That is what a time-series chart is for. It turns the thousands of pieces of valuable information onto a picture that tells a story – without throwing the information away in the process. We just need to learn how to interpret the pictures.
<Leslie> Wow! Now I understand much better why you insist we ‘plot the dots’ first.
<Bob> And now you understand the Ratio Hazards a bit better too.
<Leslie> Indeed so. And once again I have much to ponder on. Thank you again Bob.