{"id":4457,"date":"2016-01-09T11:05:47","date_gmt":"2016-01-09T10:05:47","guid":{"rendered":"http:\/\/www.saasoft.com\/blog\/?p=4457"},"modified":"2016-01-09T11:05:47","modified_gmt":"2016-01-09T10:05:47","slug":"meaningless-two-point-comparisons","status":"publish","type":"post","link":"https:\/\/hcse.blog\/?p=4457","title":{"rendered":"The Two-Points-In-Time Comparison Trap"},"content":{"rendered":"

\"comparing_information_anim_5545\"<\/a>[Bzzzzzz]<\/strong> Bob’s phone vibrated to remind him it was time for the\u00a0regular ISP remote\u00a0coaching session with Leslie. He flipped the lid of his laptop just as Leslie joined the virtual meeting.<\/p>\n

<Leslie><\/em> Hi Bob, and Happy New Year!<\/p>\n

<Bob><\/em>\u00a0Hello\u00a0Leslie and\u00a0I wish you\u00a0well in 2016\u00a0too.\u00a0 So, what shall we talk about today?<\/p>\n

<Leslie><\/em> Well, given the time of year I suppose it should be the Winter Crisis.\u00a0 The regularly repeating annual winter crisis. The one that\u00a0feels more like the perpetual winter crisis.<\/p>\n

<Bob><\/em> OK. What specifically would you like to explore?<\/p>\n

<Leslie><\/em> Specifically? The habit of comparing of this year with last year\u00a0to answer the burning question\u00a0“Are we doing better, the same or worse?”\u00a0<\/em> Especially given the\u00a0enormous effort and political attention that has been focused\u00a0on the\u00a0hot\u00a0potato\u00a0of A&E 4-hour performance.<\/p>\n

<Bob><\/em> Aaaaah! That\u00a0old chestnut! Two-Points-In-Time comparison.<\/p>\n

<Leslie><\/em>\u00a0Yes. I seem to recall you usually add the word ‘meaningless’ to that phrase.<\/p>\n

<Bob><\/em>\u00a0H’mm.\u00a0 Yes.\u00a0 It can certainly become\u00a0that, but there is a perfectly good reason why we do this.<\/p>\n

<Leslie><\/em>\u00a0Indeed, it is because we see seasonal cycles in the data so\u00a0we only want to compare the same parts of the\u00a0seasonal cycle with each other. The apples and oranges thing.<\/p>\n

<Bob><\/em> Yes, that is part\u00a0of it. So what\u00a0do you\u00a0feel is the problem?<\/p>\n

<Leslie><\/em> It feels like a lottery!\u00a0 It feels like\u00a0whether we\u00a0appear to be better or worse is\u00a0just the outcome of a random toss.<\/p>\n

<Bob><\/em> Ah!\u00a0 So we are back to the\u00a0question “Is the\u00a0variation I am looking at\u00a0signal or noise?”\u00a0<\/em><\/p>\n

<Leslie><\/em> Yes, exactly.<\/p>\n

<Bob><\/em> And we need a scientifically robust way to answer it. One that we can all trust.<\/p>\n

<Leslie><\/em> Yes.<\/p>\n

<Bob><\/em> So how do you decide that now in your improvement work? \u00a0How do you do it when you have data that does not show a seasonal cycle?<\/p>\n

<Leslie><\/em>\u00a0I plot-the-dots\u00a0and use an XmR chart to alert me to\u00a0the presence of the signals I am interested in\u00a0– especially a change of the mean.<\/p>\n

<Bob><\/em> Good.\u00a0 So why can we not use that approach here?<\/p>\n

<Leslie><\/em> Because the seasonal cycle is usually a big signal and it can swamp the smaller change I am looking for.<\/p>\n

<Bob><\/em> Exactly so. Which is why we have to abandon the XmR chart and fall back the two points in time comparison?<\/p>\n

<Leslie><\/em> That is\u00a0what I see. That is the argument I am presented with and I have no answer.<\/p>\n

<Bob><\/em> OK. It is important to appreciate that the XmR chart was not designed\u00a0for\u00a0doing this.\u00a0 It was designed\u00a0for monitoring the output quality of a stable and capable process. It was\u00a0designed to look for early warning signs; small\u00a0but significant signals that\u00a0suggest future problems. The purpose is to alert us so that we can identify the root causes, correct them\u00a0and the avoid a future problem.<\/p>\n

<Leslie><\/em> So we are using the wrong tool for the job. I sort of knew that.\u00a0But surely there must be a better way than a two-points-in-time comparison!<\/p>\n

<Bob><\/em> There\u00a0is, but first we need to understand why a TPIT\u00a0is a poor design.<\/p>\n

<Leslie><\/em> Excellent. I’m all ears.<\/p>\n

<Bob><\/em> A two point comparison is looking at the difference between two values, and that difference can be positive, zero or negative.\u00a0 In fact, it is very unlikely to be zero because\u00a0noise is always present.<\/p>\n

<Leslie><\/em> OK.<\/p>\n

<Bob><\/em> Now, both of the values we are comparing are single samples from\u00a0two bigger pools of data.\u00a0 It is the difference between the pools that we are interested in but we only have single samples of each one … so they are not measurements …\u00a0they are estimates.<\/p>\n

<Leslie><\/em> So, when we do a\u00a0TPIT comparison we are looking at the difference between two samples that come from two pools that\u00a0have inherent variation and may or may not actually be different.<\/p>\n

<Bob><\/em> Well put.\u00a0\u00a0We\u00a0give that inherent variation\u00a0a name … we call it variance\u00a0<\/strong>… and we can quantify it.<\/p>\n

<Leslie><\/em> So\u00a0if we do many TPIT comparisons then they will show variation as well … for two reasons;\u00a0first because the pools we are sampling have inherent variation; and second just from the process of sampling itself.\u00a0 It was the first lesson in the ISP-1 course.<\/p>\n

<Bob><\/em> Well done!\u00a0 So\u00a0the question is:\u00a0“How does the variance of the\u00a0TPIT sample compare with the variance of the pools that\u00a0the samples are taken from?”<\/em><\/p>\n

<Leslie><\/em> My intuition tells me that it will be less because we are subtracting.<\/p>\n

<Bob><\/em> Your intuition is half-right.\u00a0 The effect of the\u00a0variation caused by the signal will be less … that is the rationale for the TPIT after all … but the\u00a0same does not hold for the noise.<\/p>\n

<Leslie><\/em> So the noise variation in the TPIT is the same?<\/p>\n

<Bob><\/em> No. It is increased.<\/p>\n

<Leslie><\/em> What! But that\u00a0would\u00a0imply that when we do this\u00a0we are less likely\u00a0to be able to detect a change because a small shift in signal will be swamped by the increase in the noise!<\/p>\n

<Bob><\/em> Precisely.\u00a0 And the degree that the variance<\/strong> increases by is mathematically predictable … it is increased by a factor of\u00a0two.<\/p>\n

<Leslie><\/em> So as\u00a0we usually present\u00a0variation as the square root of the variance, to get it into the same units as the metric, then that will be increased\u00a0by\u00a0the square root of two … 1.414<\/p>\n

<Bob><\/em> Yes.<\/p>\n

<Leslie><\/em> I need to put this counter-intuitive theory to the test!<\/p>\n

<Bob><\/em> Excellent. Accept nothing on faith. Always test assumptions. And how will you do that?<\/p>\n

<Leslie><\/em> I will\u00a0use Excel to generate a big series of normally distributed random numbers; then I will calculate a series of TPIT differences using a fixed time interval; then I will calculate the means and variations of the two sets of data; and then I will compare them.<\/p>\n

<Bob><\/em> Excellent.\u00a0 Let us reconvene in ten minutes when you have done that.<\/p>\n

\n

10 minutes later …<\/strong><\/p>\n

\n

<Leslie><\/em> Hi Bob, OK I am ready and I would like to present the results as charts. Is that OK?<\/p>\n

<Bob><\/em> Perfect!<\/p>\n

<Leslie><\/em> Here is the first one.\u00a0 I used our A&E\u00a0performance data to give me some context. We know that on\u00a0Mondays we have an average of 210 arrivals with an approximately normal distribution and a standard deviation of 44; so I used these values to generate the random numbers. Here is the simulated Monday Arrivals chart for two years.<\/p>\n

\"TPIT_SourceData\"<\/a><\/p>\n

<Bob><\/em> OK. It looks stable as we would expect and I see that you have plotted the sigma levels which look to be just under 50 wide.<\/p>\n

<Leslie><\/em> Yes, it shows that my simulation is working.\u00a0So next is the chart of the\u00a0comparison of arrivals for\u00a0each\u00a0Monday in Year 2 compared with\u00a0the corresponding week in Year 1.<\/p>\n

\"TPIT_DifferenceData\"<\/a>\u00a0<Bob><\/em> Oooookaaaaay. What have we here?\u00a0 Another stable chart with a mean of about zero. That is what we would expect given that there has not been a change in the average from Year 1 to Year 2. And the variation has increased … sigma looks to be\u00a0just over 60.<\/p>\n

<Leslie><\/em> Yes!\u00a0 Just as the theory predicted.\u00a0 And this is not a spurious answer. I ran the simulation dozens of times and the effect is\u00a0consistent!\u00a0 So,\u00a0I am forced by\u00a0reality to accept the\u00a0conclusion that when we do two-point-in-time comparisons to eliminate\u00a0a cyclical signal we will reduce the sensitivity\u00a0of our test and make it harder to detect\u00a0other signals.<\/p>\n

<Bob><\/em>\u00a0Good work Leslie!\u00a0 Now that you have demonstrated this to yourself using a\u00a0carefully designed and conducted simulation experiment, you will be better able to\u00a0explain it to others.<\/p>\n

<Leslie><\/em> So how do we avoid this problem?<\/p>\n

<Bob><\/em> An excellent question and one that I will ask you to ponder on until our next chat.\u00a0 You know the answer to this …\u00a0you just need to bring it to conscious awareness.<\/p>\n

\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

[Bzzzzzz] Bob’s phone vibrated to remind him it was time for the\u00a0regular ISP remote\u00a0coaching session with Leslie. He flipped the lid of his laptop just as Leslie joined the virtual meeting. <Leslie> Hi Bob, and Happy New Year! <Bob>\u00a0Hello\u00a0Leslie and\u00a0I wish you\u00a0well in 2016\u00a0too.\u00a0 So, what shall we talk about today? <Leslie> Well, given the … <\/p>\n

Continue reading “The Two-Points-In-Time Comparison Trap”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,8,17,22,24,25,26,42,43,45,46,48],"tags":[],"class_list":["post-4457","post","type-post","status-publish","format-standard","hentry","category-baseline","category-bobles","category-examples","category-healthcare","category-improvementology","category-information","category-isp","category-how","category-why","category-what","category-teach","category-trust"],"_links":{"self":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/posts\/4457","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4457"}],"version-history":[{"count":0,"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/posts\/4457\/revisions"}],"wp:attachment":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4457"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4457"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4457"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}