Showing posts with label Optimization. Show all posts
Showing posts with label Optimization. Show all posts

Monday, 29 September 2014

Next Generation Point of Sale Analytics

Over the last few months I've been exploring the features I want to see in a next generation platform for point of sale analytics: It's simpler, faster and cheaper, supports rapid blending of new data sources and is powered up with real analytic capability. Looking back there are a lot of posts on this topic so here is a quick summary with links back to the detail.


Note
  • I have no immediate plans to build such a system for sale but I do use systems with many of these features for ad-hoc analytics as they are flexible yet relatively easy to set up and tear-down without incurring substantial overheads. Consider this series more of a manifesto/buyers-guide.
  • I do see changes in the marketplace suggesting that a number of DSR vendors are at least considering a move in this direction. As to which one will get there first, I think it will be whoever feels least weighed down by their existing architecture.
Database technology has moved on dramatically over the last few years. For this scale of data, analytic solutions should be columnar, parallel and (possibly) in memory. This enables speed, scalability and a simple data structure that makes it easy to hook up whatever analytic or BI tools you wish.
If the only data you have in the system is pos sales for a single retailer, you can build a reporting system ("what sold well last week") but you will struggle to understand why sales change. Bringing in other data sources: multi-retailer, demographics, weather information, promotional calendars, competitor activity, socio-economic trends, Google trends, social media, etc. allow for much more insighful analtyics. It's not easy to do this though, particularly if your source database is locked down so that it takes a software engineer to add tables
The term "Analytics" in general use covers a lot of activities most of which involve little more than reporting. In some instances you can slice and dice your way through a dataset to find insight, reporting is not without value but it's not analytics. Not even close.
Can you buy good analytics? Yes, but there are also a number of pseudo-analytic solutions in the market that have little to no analytic power - caveat emptor!
To get to real, deep insights you need real analytic tools. Depending on the taxonomy you are used to, we are talking about predictive and prescriptive analytics,machine learning, statistics, optimization or data science. Most of these tools are not new but they are not generally found in standard BI offerings and even when they are (e.g. reporting level R integration) you may struggle to apply the analytic tools at scale.
Finally, whether you build your own analytic tools or buy them in to run on your platform, clever math is not enough. If a user cannot comprehend the tool or it's suggestions due to poor user interface design and /or bad visualization choices it's worth precisely ... squat.

Monday, 4 August 2014

Next Generation DSRs - An Analytic name is not enough

You need not always build your analytic tools, sometimes you should buy in. If the chosen application does what you need that often makes good economic sense... as long as you know what you are buying.

Let's be clear, an Analytic name does NOT mean there are any real Analytics under the hood.

For many managers, Analytics is akin to magic. They do not know how an analytics application works in a meaningful way and have no real interest in knowing. At the same time, there is no business standard for what makes up "forecasting", "inventory optimization", "cluster analysis", "pricing analysis", "shopper analytics", "like products" or even (my favorite) "optimization".  Don't buy a lemon!


In the worst examples, there is nothing under the hood at all. One promotion-analytic tool I came across recently proudly proclaimed that you (the user) could calculate the baseline and lift for each promotion however you saw fit and then just enter the result into their system. They presented this as a positive feature, but calculating a meaningful baseline and lift is the difficult part!!

I've seen similar approaches for:
  • off-shelf alerting tools that ask you how long of a period of zero sales is abnormal (so they can report exceptions)
  • supply chain systems that need you to enter safety-stocks or re-order-points (so they can figure out when to order).  
  • assortment optimization tools that want you to input product substitution rates.
Hmmm, is a car without an engine still a car?
Many applications use pseudo-analytics. After all, how hard can it be? "cluster analysis" , that's finding groups of things right? I reckon I can figure that out, no stats required. Yeah, right, of course you can... FYI - meaningful, useful clusters may be a little more difficult. It's not that cluster analysis is particularly hard, but neither is it something you can knock together without the right tools or any statistical understanding.
Sadly, I have seen real world examples of pseudo-analytics too in pricing analytics, off-shelf alerting, demographic analyses, inventory optimization and forecasting.
The right tool for the right job. There are many good analytic applications available, but you can still make it useless if it does not suit the task you have in mind. Using a time-bucket oriented optimization program to schedule production runs with sequencing comes to mind. OK, relatively few people are going to understand that one and it's not a DSR application, but it is real, the software vendor did not come out shouting that there would be a problem and 2 years down the line that project was abandoned.

Are DSRs worse than other applications?

I think this kind of feature-optimism, is a general issue in buying any analytic app but my perception is that it is a bigger problem in the DSR space.  Perhaps because the DSR is trying to offer so much analytic functionality to so many functional areas?  Is a DSR really going to handle forecasting, pricing-analytics, cluster-analysis, weather-sensitivity-modeling, promotional analytics, inventory optimization, assortment selection and demographic analysis (note - not a complete list), all as packaged software, for $50K a year?   Not unless they can scale that investment across a huge user-base.  Some will be good, others not so much - be warned.   

Spotting a lemon

An expert in the field (with analytic and domain knowledge) can spot a lemon from quite a distance. If you do not possess one you would be wise to invest in some consulting to bolster your purchasing team. For those applications that pass the sniff-test, the proof of any analytic system is in it's performance. Define rational performance criteria, test, validate, pilot and never, ever, ever rely on a software vendor ticking the box in your RFP.

Monday, 4 February 2013

Business Analytics - The Right Tools For The Job

Whether your analytic tool of choice is Excel or R or Access or SQL Server or ... whatever,  if you've worked a reasonable range of analytic problems I will guarantee that at some point you have tried to make your preferred tool do a job it is not intended for or that it is ill-suited for.  The end result is an error-prone, maintenance nightmare and there is a better way.

We all know when we are pushing it too far - the system starts to "creak".  Symptoms vary but include at least some of the following:
  • Model calculations run slowly (if they complete at all).
  • Model calculations give, apparently, inconsistent results.
  • Making minor changes becomes a major headache
  • Errors ("bugs") are routine.  When you need to do a demo, you pray first.
  • You have learned to avoid certain operations because the likelihood of success is slim.
  • If you must come back to your model/application after 6 months to update it, you feel physically ill.
  • and perhaps most importantly, you are far from sure that using your model generates the right results.
This isn't just an academic issue or one of preference.  Poor models can cost real money in lost opportunity or bad decisions that get implemented.  (See this post for a few examples)

So why do we do this ?  I think it's because for many analysts their tool box looks like this.

Why is their toolbox so empty?  In some cases, corporate IT restrictions may make it very difficult to  acquire/install the right tools;  I've been there, it's a real challenge.

For many people though, it just feels easier to take a tool they know well and try and "make it work" than to learn something new.  That's rather like thinking  "I need to chop down a tree... I'll sharpen my hammer" 

Last week, I gave you my nominations for "The Worst use of Excel Ever!"  I could easily cite similar abuses for other tools and over the next few months I will.   

This blog post is the first in a series around using "The Right Tool For The Job".   I'm going to encourage you to add a few more tools to your analytic tool-box and learn to wield them effectively. (Or, alternatively, to at least recognize when you need someone who can do that for you.)  Do you need: 
  • A application programming environment ? 
  • A database (of varying capability, Access, SQL or perhaps a newer column storage databases) ?
  • A reporting tool ?
  • An optimization modeling language ?
  • A visualization tool ?
  • A statistics or data-mining package ?
  • A simulation package ?
Which tools do you think are the most mis-used?  What skills/tools should a business analyst consider adding to their toolbox ?  

Monday, 21 January 2013

Recommended Reading: Supply Chain Network Design


I've done a lot of  supply chain network design projects and consider myself to be an expert. Had I had this book from the start, I may have got to expert status a lot faster.

With experience in supply-chain and an academic background that includes mathematical-optimization, when the need arose to build supply chain network optimization models I just did it.  Then I learned many, valuable, real-world lessons the hard way- by getting it wrong.

There are a number of books available that cover this area: I have dipped into a few, as needed, and I have not read most of them so I really can't say this is the best book available on the subject.  I can say this is one of the very few analytic books on any subject that I have read cover to cover.  


Network design is perhaps not as hot a topic now as it was 10 years ago.  That's just my perception, but while the hype right now is around "big data", network design continues to deliver major savings to organizations.  Network design finds where your facilities should be and how product should flow through them to support your business at the lowest cost.  The more rapidly your business is changing the more often this is worthwhile: an acquisition or divestment will almost always justify the expense with a significant ROI.  A 10% reduction in supply chain cost is common..  Even on a stable business there can be significant saving (transportation, labor and warehousing) in adjusting product flow on a relatively frequent, annual basis.

Note that while the authors (all from IBM) have extensive experience building software products to help you do supply-chain network optimization this book is not a sales brochure for LogicNet, in fact, it's barely mentioned.

The math needed to run an optimization model is not simple but it is accessible to those who want to learn and this book does take you through step by step a mathematical programming model that gets increasingly sophisticated.  The necessary theory is all there.

What attracted me was that it goes beyond the theory and has lots of details around project execution:.  the need for sensitivity analysis; the difficulty of getting reliable transportation rates ; sensible data aggregation strategies; why you must have an optimized "baseline"; and numerous others.  These are all areas that analysts get wrong - as I did.  Some learn from the experience, others send out the results anyway.

Managers who hope to become better buyers/consumers of network-design projects (remembering that  your analysts may also be making newbie mistakes) skip the math sections and you can still understand what can be modeled and why you would want to.

For analysts actively involved in building optimization-models the mathematical formulations are extremely helpful. Even if you choose to build models with a software package that tries to hide the harder math from you, the guidance around data and the art of modeling is worth the price and the time to read it.

If you have a network design project in mind and a plane journey coming up - make the investment.  

Supply Chain Network Design: Applying Optimization and Analytics to the Global Supply Chain (FT Press Operations Management) by Michael Watson, Sara Lewis, Peter Cacioppi and Jay Jayaraman (Sep 1, 20112)



Monday, 29 October 2012

Truckload Transportation - are you paying to ship air ?



How full is a "full" truck?   Not sure?  That's a shame, because when you contract for truckload freight, you pay for the whole vehicle, whether you fill it or not.   As I'll show you, the regulations around what constitutes "full" for weight are very complex.  In addition, the 3D jigsaw puzzle to pack product into the trailer space, distributing weight correctly and minimizing damage is exceptionally challenging.  Get it wrong and you are paying to ship air.  

It is much easier to plan to approximate rules or guidelines than to figure out what's really going on and it is common in the CPG industry to plan transportation loads based on these approximate rules.   Unfortunately, these approximate rules are very dependent on what you are trying to load (as well as who made up the "rule") so you will rarely encounter the same "rule" twice.  Typically what you will see is based on weight, pallet positions or cube.  Rules that restrict what's loaded to maximum limits like:

  • 40,000 lbs of product
  • 2,500 cubic ft of product
  • 48 pallets of product.
None of these are right and they routinely result in shipping air.

Depending on the product and the vehicle,  I can safely, and legally, load much more than 40,000 lbs of product, (much) more than 2,500 cubic feet many more than 48 pallets.  

One particularly bad example I have encountered said "a truck is full when there is 38,000 lbs of product in it".  If I can find a way to, legally and safely load 45,000 lbs in the trailer it’s as though 7,000 lbs of product just shipped free, effectively saving 15% ((7,000/45,000 = 15.5%) in freight cost.

So, why is this so difficult to get right?  Let's look at the regulations around weight.


Weight Limits

If your product is "heavy", you will probably hit a weight limit in loading.  "Heavy" in this instance is roughly 20 lbs per cubic foot or more.  Much less than this and you will probably hit a space limit first (see below).

The government sensibly places restrictions on how heavy a loaded vehicle may be and how that weight must be distributed to be carried safely.   To get a feel for  the complexity involved, here is a link to the relevant page from the US Department of Transportation.  Stay just long enough to get confused then head back here :-)   Bridge Formula Weights 

To summarize (and simplify):
  • The  total weight of the vehicle (including product, packaging, pallets, fuel, driver, etc...) must not exceed 80,000 lbs.
  • There are limits for the weight on individual axles (shown below in this graphic from the US Department of Transport)
Note for the mathematically inclined:
If anyone is really interested in being able to calculate what weight should be on each axle given a particular layout of product in the trailer you need a little physics/engineering math.  Here is a great example of how a beam transfers load with an interactive calculator.
With the right math, you can calculate the center of gravity for the product and how that weight would be distributed to the axles.  Move the center of gravity forwards (e.g. by moving heavier product to the front) and you take weight of the rear wheels and transfer it to the tractor unit.  Quite how that weight then gets distributed to to axles 1 through 3 depends on where the "kingpin" connection (between the trailer and the tractor unit) is relative to the tractor units axles. You can build such a model in Excel.
So, can you plan to a total rig weight of 80,000 lbs?  No, sorry,  there are only so many options for how to layout product in the vehicle and there may be no layout that balances weight well enough across all axles to max out the 80,000 lb overall limit.You need to find the layout that gets closest to that limit and live with the loss.

Alternatively, you may run out of space in the vehicle before you get close to a weight limit.

Space Limits

If product is reasonably light (low density) we will probably be constrained by space before weight becomes a problem..

A reasonably standard trailer's internal dimensions are approximately
  • 52' long
  • 8' wide
  • 8' (usable) height
That's a little over 3,300 cubic ft of space available to you.  However, you are typically loading with palletized product and you will not get to use most of this space.   Let's look at how well you can use the floor space first.


A standard pallet for US grocery is 40" wide, 48" long. The ability to fill the trailer floor-space depends a lot on how well these palletized units fit.  


If pallets will fit in the trailer "wide:wide" it's possible to put 30 pallet footprints in a 53' trailer.








 If  the trailer is too narrow or product overhangs the pallet, you have to go to other configurations.   "Chimney stacking" maxes out at 28 footprints.  



If you can't turn pallets at all, "narrow:narrow" loads max out at 26 footprints (although with any overhang on the pallets at all you will likely get 24/25)




In each case there is floor space you cannot use.  Load "narrow:narrow" and you have already lost almost 20% of the available space.


Now let's look at vertical space.  This is much more variation in the height of trailers and in the height of doors to those trailers.  If the door height is restrictive , perhaps because the door rolls up inside the trailer, that will limit the product you can get into that trailer.  Let's assume for now that we can safely get to 8' high. 

How much of the vertical space we can use depends on what we are loading.  Some product cannot be stacked, pallet on pallet, without causing damage.  Some pallets are too tall to allow anything (except an unusually short pallet) to fit stacked on top of it.  Very short pallets may be able to stack 3 or even 4 high.

If I assume 40" high palletized product, double stacked, we can use 80" of the 96" vertical space (83%).
Combine that with "narrow:narrow" loading and we max out a truck at 83% * 83% = 69% space utilization. From a "cube" standpoint the trailer is only 70% full and it's at capacity.  Hence rules like "the trailer is full at 2,300 cubic feet of product" when the trailer has over 3,300 cubic feet of air.

Depending on product dimensions and the ability to stack product we may be able to load much more than 2,300 cubic feet or much less.

Reducing Damage

We could spend a lot of time on this, but for now suffice to say, we would like to load the trailer so that product is less likely to get crushed, to move as the vehicle corners or to land in a heap at the front when the driver must, necessarily, brake.  This puts additional constraints on what product can go where in the vehicle, reducing, again, the weight you can carry and/or the volume you can load into the available space.

Equipment  limitations

If the pallet handling equipment (for either shipper or receiver) can't stack pallets or can't handle them turned, you will necessarily lose a lot of payload capacity.  Unless this is for very short trips where you can pay for the freight-cost with handling labor savings, you need to invest in warehouse equipment.

Not all trucks/trailers are the same.  By design they can be very different, tractor units weighing anywhere between 11,000 and 20,000  lbs.  Trailers can be vary by a few thousand lbs too.  Even equipment of the same make, model and year can be different depending on its setup (kingpin position) and the addition of aftermarket parts.  (Mount a new fuel tank too near the front and see it use up the limited capacity you have on the steering axle).  If weight is an issue for you, you will need to work with your carriers to understand what equipment they are bringing in.  Lightweight equipment is worth more to you, heavyweight equipment should be avoided or contracted at a low enough rate to offset the loss in carrying capacity..

Pulling it all together

Each of these sets of restrictions, weight, space, damage and equipment are complex: accurately modeling any of them is a challenge.  Depending on what you load: heavy or light product, slightly oversize, stack-able or not,  the factors that constrain you continually move.  And, beyond modeling it, you need to optimize: to find the selection and layout of product that maximizes your vehicle loading.

My Take

If you're lucky enough to have consistently-sized, palletized product with consistent low or high density that is consistently  stacked and loaded on consistent carrier equipment  you may be able to build a simple rule that really does max out your truck loading.  Good for you!   For the rest of us, such "rules" are poor guesses at best and can leave a lot of money on the table.  

You are NOT going to build this in Excel unless you have a lot of functional knowledge, very advanced skills in mathematical-optimization and the ability to program your own optimization code.  I have built a load optimization tool (It's a weakness, I like to know how things work).  I have also built my own tools for inventory-optimization, neural-network modeling, genetic-optimization, forecasting and many other needs..  Some of these tools are still in production use today, others were essentially learning opportunities.  In this instance, I chose to buy the software because it has richer functionality than what I chose to build myself.

(On a technical note, application-specific,heuristic optimization routines solve these problems well and quickly.  Mixed-Integer-Programming can get you most of the way, but personally I can't see how to embed some of the damage-reduction ideas into a linear objective function and as the heuristic works so well, you don't need the overhead or complexity of integrating a math programming tool) 

Any load building “optimizer” that asks you to specify a maximum weight or cube or number of pallets is really just automating these approximate rules rather than helping you truly max out the load.  An optimizer that is implemented as a stand-alone package is interesting but not really useful: you need this capability integrated into order-processing, deployment and transportation planning to be effective.  You need the right tool for the job.  These tools do exist, they work and it's not worth your time or effort to write them again.

There are a number of tools in the market that work in this space.  Google "load building optimizer" to see some.  I have not reviewed them all, far from it, but I can tell you that not all "optimization" is the same.  Having "optimize" in the sales literature does not mean it will do a great job for you or that any actual optimization is really taking place.

If you want a quick recommendation I suggest you talk to Transportation | Warehouse Optimization. they have a great load building tool and can extend this further into optimizing case-pick routing, pallet builds and shipping locations.

Let's assume you are doing a reasonably good job today without a load optimizer.  What would an extra 5% off your freight spend be worth to you?  Enough to invest in the right tool for the job?

A final thought

These rules-of-thumb can be very persistent.  Having implemented a system to drive increased payload coming out of one manufacturing site, I was perplexed some months later to see that the load factor had shrunk back to where it started.  It turns out that the warehouse supervisor at the plant was adjusting each and every load plan manually to fit with his interpretation of the “rules”.