{"id":2671,"date":"2013-01-26T12:54:12","date_gmt":"2013-01-26T12:54:12","guid":{"rendered":"http:\/\/www.saasoft.com\/blog\/?p=2671"},"modified":"2013-01-26T12:54:12","modified_gmt":"2013-01-26T12:54:12","slug":"curing-carveoutosis","status":"publish","type":"post","link":"https:\/\/hcse.blog\/?p=2671","title":{"rendered":"Curing Chronic Carveoutosis"},"content":{"rendered":"

\"pin_marker_lighting_up_150_wht_6683\"<\/a>Last week the Ray Of Hope briefly illuminated a very common system design disease called carveoutosis<\/em>.\u00a0 This week the RoH will\u00a0tarry a little longer to illuminate an example that reveals the\u00a0value of diagnosing and treating this endemic\u00a0process ailment.<\/p>\n

Do\u00a0you remember the days\u00a0when we used to have to visit the Central Post Office in our lunch hour to access\u00a0a quality-of-life-critical\u00a0service that only a\u00a0Central Post Office could provide – like getting a\u00a0new road tax disc for our car?\u00a0 On walking through the impressive Victorian entrances of these\u00a0stalwart high street institutions our primary\u00a0challenge was to decide which queue to join<\/strong>.<\/p>\n

In front of each gleaming mahogony, brass and glass\u00a0counter was a queue of waiting\u00a0customers. Behind was the Post Office operative.\u00a0We knew from experience that to be in-and-out before our lunch hour expired required deep understanding of the ways of people and processes – and a savvy selection.\u00a0 Some queues\u00a0were longer than others. Was that because there was a particularly\u00a0slow operative\u00a0behind that counter? Or was it because there was a\u00a0particularly\u00a0complex\u00a0postal problem being processed?\u00a0Or was it because\u00a0the customers who had been\u00a0waiting longer had identified that\u00a0queue\u00a0was fast flowing and had\u00a0defected to it from their more torpid streams? We know that size is not a reliable indicator of speed or quality.\"figure_juggling_time_150_wht_4437\"<\/a><\/p>\n

The social pressure is now mounting … we must choose … dithering is a sign of weakness … and swapping queues later is another\u00a0abhorrent behaviour.\u00a0So we\u00a0employ\u00a0our\u00a0most trusted heuristic – we join the end of the shortest queue. Sometimes it is a good choice, sometimes not so good!\u00a0 But intuitively it feels like the best option.<\/p>\n

Of course\u00a0 if we choose wisely and\u00a0we succeed in leap-frogging our fellow customers then we can swagger (just a bit) on the way out. And if not\u00a0we can scowl and mutter\u00a0oaths at others who (by sheer luck)\u00a0leap frog us. The\u00a0Post Office\u00a0Game is\u00a0fertile soil for the Aint’ It Awful<\/em> game which we play when we arrive back\u00a0at work.<\/p>\n

\"single_file_line_PA_150_wht_3113\"<\/a>But those days are past and now we are more likely to encounter a single-queue<\/strong>\u00a0when we are forced by necessity\u00a0to embark on a midday shopping sortie. As we enter we see the path of the snake\u00a0thoughtfully marked out with rope barriers\u00a0or with shelves\u00a0hopefully stacked\u00a0with\u00a0just-what-we-need bargains to stock up on as we drift past.\u00a0 We are processed FIFO (first-in-first-out) which is fairer-for-all\u00a0and avoids the challenge of the dreaded choice-of-queue. But the single-queue snake brings a new challenge: when we reach the head of the snake we must\u00a0identify which\u00a0operative\u00a0has become\u00a0available first – and quickly!<\/p>\n

Because if we falter then we will incur the shame of the finger-wagging or the flashing red neon arrow that is easily visible to the whole snake;\u00a0and\u00a0a painful\u00a0jab in the ribs from the impatient\u00a0snaker behind us;\u00a0and a\u00a0chorus of tuts from the tail of the snake. So\u00a0as we frantically scan left and right along the\u00a0line of bullet-proof glass cells\u00a0looking for clues of imminent availability we run the risk of\u00a0developing acute vertigo or a painful repetitive-strain neck injury!<\/p>\n

\"stick_figure_sitting_confused_150_wht_2587\"<\/a>So is the single-queue design better?\u00a0 Do we actually wait less time, the same time\u00a0or more time? Do we pay a fair price for fair-for-all queue design?\u00a0The answer\u00a0is not intuitively obvious because when we are forced to join a lone and long queue\u00a0it goes against\u00a0our gut instinct. We feel the urge to push.<\/p>\n

The short answer is “Yes”.\u00a0\u00a0<\/strong>A single-queue feeding\u00a0tasks to\u00a0parallel-servers is actually a better design.\u00a0And if we ask the\u00a0Queue Theorists then they will dazzle us with complex equations that\u00a0prove<\/strong> it is a better design – in theory.\u00a0\u00a0But the scary-maths does not help us to understand how<\/em> it is a better design.\u00a0Most of us are not able to convert\u00a0equations into experience; academic rhetoric into pragmatic reality. We need to see it with our own eyes to know it and understand it. Because we know that reality is messier than theory.\u00a0\u00a0\u00a0\u00a0<\/p>\n

And if it is a better design\u00a0then just how much better is it?<\/p>\n

To illustrate the\u00a0potential advantage\u00a0of a single-queue design we need to\u00a0push the competing candiates\u00a0to their performance limits and then measure the difference. We need a real example and some real data. We\u00a0are Improvementologists!\u00a0<\/p>\n

First\u00a0we need to map<\/strong> our Post Office process – and that reveals that we have a single step process – just the counter. That is about as simple as a process gets. Our map also shows that we have a row of counters of which five are manned by fully trained\u00a0Post Office service operatives.<\/p>\n

\"stick_figure_run_clock_150_wht_7094\"<\/a>Now we can measure<\/strong> our\u00a0process and when we do that we find that we get an average of 30 customers\u00a0per hour walking in the\u00a0entrance and and average of 30 cusomers an hour walking out. Flow-out equals flow-in. Activity equals demand. And the average flow\u00a0is one every 2 minutes. So far so good. We then\u00a0observe our five operatives and we find that the average\u00a0time from starting to serve one customer\u00a0to starting to serve the next is 10 minutes. We know from our IS training that this is the cycle time. Good.<\/p>\n

So we do a quick napkin calculation to check and that the numbers make sense: our system of five operatives working in parallel, each with an average\u00a0cycle time of 10 minutes can collectively process a customer on average every 2 minutes – that is 30 per hour on\u00a0average. So it appears we have just enough capacity to keep up with the flow of work\u00a0\u00a0– we are at the limit of efficiency.\u00a0 Good.<\/p>\n

\"CarveOut_00\"<\/a>We also notice that there is variation in the cycle time from customer to customer – so we plot our individual measurements asa time-series chart. There does not seem to be an obvious pattern – it looks random – and\u00a0BaseLine says that it is statistically stable. Our chart tells us that\u00a0a range of 5\u00a0to 15 minutes\u00a0is a\u00a0reasonable expectation to set.<\/p>\n

We also observe that there\u00a0is always a queue of waiting customers somewhere – and although the queues fluctuate in size and location they are always there.<\/p>\n

\u00a0So there is always a wait for some customers. A variable wait;\u00a0an unpredictable wait. And that is a concern\u00a0for us because when the queues are too numerous and too long then we see customers get agitated, look at their watches, shrug their shoulders and leave – taking their custom and our\u00a0income with them and no doubt telling all their friends of their poor experience.\u00a0Long queues and long waits are bad<\/strong> for business.<\/p>\n

And\u00a0we do not want zero\u00a0queues either because if there is no queue and our operatives run out of work\u00a0then they become\u00a0under-utilised\u00a0and our system efficiency and productivity falls.\u00a0\u00a0That means we are incurring a cost but\u00a0not generating an income.\u00a0No queues and idle resources\u00a0are bad<\/strong> for business too.<\/p>\n

And we do not want a mixture of quick queues and slow queues because that causes complaints and conflict.\u00a0 A high-conflict customer complaint experience is bad<\/strong> for business too!\u00a0<\/p>\n

What we want is a design that\u00a0creates small<\/strong> and stable<\/strong> queues; ones that are just big enough to\u00a0keep our operatives\u00a0busy and our customers not waiting too long.<\/p>\n

So which is the better design and how much better is it? Five-queues or a single-queue? Carve-out or no-carve-out?<\/em><\/p>\n

To\u00a0find the answer we decide to conduct a week-long series of experiments on our system and use real data\u00a0to\u00a0reveal the answer. We choose the time from a customer arriving to\u00a0the same\u00a0customer leaving as our measure of quality and performance – and we know that the best we can expect is somewhere between 5 and 15 minutes.\u00a0 We know from our IS training that is called the Lead Time.<\/p>\n

\"time_moving_fast_150_wht_10108\"<\/a>On day\u00a0#1<\/strong> we arrange\u00a0our Post Office\u00a0with five queues – clearly roped out – one for each manned counter.\u00a0 We know\u00a0from our mapping and measuring that customers do not arrive in a steady stream and we fear that may confound our experiment so we arrange to admit only one\u00a0of our loyal and willing customers every 2 minutes. We also\u00a0advise our loyal and willing customers which queue they must join before they enter to avoid the\u00a0customer choice challenges.\u00a0\u00a0We decide\u00a0which queue\u00a0using\u00a0a random number generator – we toss a dice until we get a number between 1 and 5.\u00a0 We record the time the customer\u00a0enters on a slip of paper and we ask the customer to give it to the operative and\u00a0we instruct our service\u00a0operatives to record the time they completed their work on the same slip and keep it for us to analyse later.\u00a0We run the experiment for only\u00a01 hour so that we have a sample of\u00a030\u00a0slips and then we collect the slips,\u00a0\u00a0calculate the difference between the arrival and departure times and plot them on a time-series chart in the order of arrival.<\/p>\n

\"CarveOut_01\"<\/a>This is what we found.\u00a0 Given that the time at the counter is an average of 10 minutes then some of these lead times seem quite long. Some customers spend more time waiting than being served. And we sense that the performance is getting worse over time.<\/p>\n

So for the next experiment we decide to open a sixth counter and\u00a0to rope off a sixth queue. We expect that increasing capacity will reduce waiting time and\u00a0we confidently expect the performance to improve.<\/p>\n

On day #2<\/strong> we run our experiment again, letting customers in one every 2 minutes as before and this time we use all the numbers on the dice to decide which queue to direct\u00a0each customer to.\u00a0 At the end of the\u00a0hour we collect the slips, calculate the lead times and plot the\u00a0data – on the same chart.<\/p>\n

\"CarveOut_02\"<\/a>This is what we see.<\/p>\n

It does not look much better and that is big surprise!<\/p>\n

The wide variation\u00a0from customer to customer looks about the same\u00a0but with the Eye of Optimism\u00a0we get a sense that the overall performance looks a bit\u00a0more stable.<\/p>\n

So we conclude that adding capacity (and cost) may make a small difference.<\/p>\n

But then we remember that we still only served 30 customers – which means that\u00a0our\u00a0income stayed the same while\u00a0our\u00a0cost increased by 20%.\u00a0That is definitely NOT<\/strong> good for business: it is not goiug to look good in a business case “possible marginally better quality and 20% increase in cost and therefore\u00a0price<\/em>!”<\/p>\n

So on day #3<\/strong> we change the layout. This time we go back to\u00a0five\u00a0counters\u00a0but we re-arrange the ropes to create a single-queue so the customer at the front\u00a0can be ‘pulled’ to the first available counter.\u00a0Everything else stays the same – one customer arriving every 2 minutes, the dice, the slips of paper, everything.\u00a0 At the end of the\u00a0hour we collect the slips, do\u00a0our sums\u00a0and plot\u00a0our chart.<\/p>\n

\"CarveOut_03\"<\/a>And this is what we get!\u00a0The improvement is\u00a0dramatic. Both the average and the variation has fallen –\u00a0especially the variation.\u00a0But surely this cannot be right. The improvement is too good to be true.\u00a0We check\u00a0our data again.\u00a0Yes, our customers arrived and departed on average one every 2 minutes as before; and all our operatives did the work in an average of 10 minutes just as before. And we had the\u00a0exactly the same capacity as we had on day #1. And we finished on time.\u00a0It is correct. We are gobsmaked. It is like a magic wand has been waved over our process. We never would have predicted \u00a0that just<\/strong> moving the ropes around to could have such a big impact.\u00a0 The Queue Theorists were correct after all!<\/p>\n

But wait a minute! We are delivering a much better customer experience in terms of waiting time and at the same cost. So could we do even better with six\u00a0counters open? What will happen if we keep the single-queue design and\u00a0open the sixth desk?\u00a0 Before it made little difference but now we doubt our ability to guess what will happen. Our intuition seems to keep tricking us. We\u00a0are losing our\u00a0confidence in predicting what the\u00a0impact will\u00a0be. We are in counter-intuitive land! We need to run the experiment for real.<\/p>\n

So on day #4<\/strong> we keep the single-queue and we\u00a0open six desks.\u00a0We await the data eagerly.<\/p>\n

\"CarveOut_04\"<\/a>And this is what happened. Increasing the capacity\u00a0by 20% has made virtually no difference – again. So we\u00a0now have two pieces of evidence that say – adding extra capacity did\u00a0not make a difference to waiting times. The variation looks a bit less though but it is marginal.<\/p>\n

It was changing the Queue Design that made the difference! And that change cost nothing. Rien. Nada. Zippo!<\/p>\n

That will look much better\u00a0in our report but now we have\u00a0to face the emotional discomfort of having to re-evaluate one of our deepest held assumptions.<\/p>\n

Reality is telling us that we are delivering a\u00a0better quality experience using exactly the same resources and it cost nothing to achieve.\u00a0Higher quality did NOT<\/strong> cost more. In fact we can see that with a carve-out design when we added capacity we just increased the cost we did\u00a0NOT<\/strong> improve quality. Wow!\u00a0 That is a shock. Everything we have been led to believe seems to be flawed.<\/p>\n

Our senior managers are not going to like this message at all! We will be challening\u00a0their dogma directly. And they do not like that.\u00a0Oh dear!\u00a0<\/p>\n

Now we can see how much better a no-carveout single-queue pull-design can work; and now we can explain why\u00a0single-queue designs\u00a0\u00a0are used; and now we can\u00a0show others our experiment and our\u00a0data and if they do not believe us they can repeat the experiment themselves.\u00a0 And we can see that it does not need a real Post Office – a pad of Post It\u00ae Notes, a few stopwatches and some willing helpers\u00a0is all we need.<\/p>\n

And even though we have seen it with our own eyes we still struggle to explain how<\/strong>\u00a0the single-queue design works better. What actually happens? And we still have that niggling feeling that the performance on day #1 was unstable.\u00a0 We need to do some more exploring.<\/p>\n

So we run the day#1 experiment again – the five queues –\u00a0but this time we\u00a0run it\u00a0for a whole day, not just an hour.<\/p>\n

\"CarveOut_06\"<\/a><\/p>\n

Ah ha!\u00a0\u00a0 Our hunch was right.\u00a0\u00a0It is an unstable design. Over time the variation gets bigger and bigger.<\/p>\n

But how can that happen?<\/p>\n

Then we remember. We told the customers that they could not<\/strong> choose the shortest queue or change queue after they had joined it.\u00a0 In effect\u00a0we said “do not look at the other queues<\/em>“.<\/p>\n

And that happens all the time on our systems when we jealously hide performance data from each other! If we are seen to\u00a0have a smaller queue we get given extra work\u00a0by the management or told to slow down by the union rep!\u00a0\u00a0<\/p>\n

So what do we do now?\u00a0 All we are doing is trying to improve the service and all we seem to be achieving is annoying more and more people.<\/p>\n

What if we apply\u00a0a maximum waiting time target, say of 1 hour, and allow customers\u00a0to jump to the front of their queue if they\u00a0are at risk if breaching the target? That will smooth out spikes and give everyone a fair chance. Customers will understand. It is intuitively obvious and common sense. But our intuition has tricked us before …\u00a0<\/p>\n

So we run the experiment again and this time we tell our customers that if they wait 50 minutes then they\u00a0can jump to the front of their queue. They appreciate this because they now have a upper limit on the time they will wait.\u00a0\u00a0<\/p>\n

\"CarveOut_07\"<\/a>And\u00a0this is what we observe. It looks better than before, at least initially, and then it goes pear-shaped.<\/p>\n

All we have done with our\u00a0‘carve-out and-expedite-the-long-waiters’\u00a0<\/em>design is to defer the inevitable – the crunch. We cannot keep our promise. By the end everyone is pushing to the frontof the queue. It is a riot!\u00a0\u00a0<\/p>\n

And there is more. Look at the lead time for the last\u00a0few customers – two hours. Not only have they waited a long time, but we have had to stay open for two hours longer. That is a BIG cost pessure in overtime payments.<\/p>\n

So, whatever way we look at it: a single-queue design is better.\u00a0 And no one loses out! The customers have a short and predictable waiting time; the operatives are kept occupied and go home on time; and the executives bask in the reflected glory of the excellent customer feedback.\u00a0\u00a0It is a Three Wins\u00ae design.<\/p>\n

Seeing is believing – and\u00a0we now know that it is worth diagnosing and treating carveoutosis<\/em>.<\/p>\n

And the only thing left to do is to explain is how<\/strong> a single-queue design works better. It is not obvious is it?\u00a0<\/p>\n

\"puzzle_lightbulb_build_PA_150_wht_4587\"<\/a>And the best way to do that is to play the Post Office Game and see what actually happens.\u00a0<\/p>\n

A big light-bulb moment awaits!<\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

Update:<\/span> My little Sylvanian friends have tried the Post Office Game and kindly sent me this video of the before<\/em>\u00a0\u00a0Sylvanian Post Office Before<\/a>\u00a0and the after Sylvanian Post Office After<\/a>. They say they now know how the single-queue design works better.<\/em>\u00a0<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Last week the Ray Of Hope briefly illuminated a very common system design disease called carveoutosis.\u00a0 This week the RoH will\u00a0tarry a little longer to illuminate an example that reveals the\u00a0value of diagnosing and treating this endemic\u00a0process ailment. Do\u00a0you remember the days\u00a0when we used to have to visit the Central Post Office in our lunch … <\/p>\n

Continue reading “Curing Chronic Carveoutosis”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,10,12,17,18,20,30,32,33,42,43,44,45,48],"tags":[],"class_list":["post-2671","post","type-post","status-publish","format-standard","hentry","category-6m-design","category-business","category-carveout","category-examples","category-finance","category-flow","category-operations","category-productivity","category-quality","category-how","category-why","category-three-wins-r","category-what","category-trust"],"_links":{"self":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/posts\/2671","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2671"}],"version-history":[{"count":0,"href":"https:\/\/hcse.blog\/index.php?rest_route=\/wp\/v2\/posts\/2671\/revisions"}],"wp:attachment":[{"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2671"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2671"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hcse.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2671"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}