Best Answers to Commonly Asked Risk Analysis Questions

Many managers don’t understand what exactly risk analysis is.  We put together some of the most common questions with responses for you.

What does the risk percentage mean?

The risk percentage approximates the on-time probability for an order with appropriate consideration of the number of replications or “experiments.”  It tells the user how confident they can be in meeting the due date given how many trials they have conducted. 

How does Simio calculate the on-time probability?

Simio adjusts from a base rate of 50% with each risk replication.  If an order is on time in an individual replication, Simio updates the probability, increasing it closer to 100%.  If the order is late, Simio decreases the probability closer to 0%.  Each replication is an experiment that provides new information about the likelihood of success or failure.  More experiments mean more confidence in the answer.

Why is the base rate 50%?

Before any plan is generated or any activity is simulated, there is no information about the order other than the possible outcomes.  Because there are only two outcomes that matter (on time or not), the base rate is set to 50%.

I have an overdue order in my system.  Why is it not always 0%?

Because the calculation is an adjustment of a base rate of 50%, Simio needs a lot of evidence before it will guarantee that an order will be late (or on time for that matter).  If the user runs 1000 replications, and the result is late in all of them, Simio will reflect a 0% on time probability. 

What formula does Simio use to calculate the probability?

For the statistics experts, Simio uses a binomial proportion confidence internal formula known as the Wilson Score.  We report the midpoint of the confidence interval as the risk measure.

Why not just report the outcome of the replications as the probability (e.g., if 9 of 10 are on time, report 90% on time probability)?

This was the original implementation.  However, it gives a false sense of confidence and can be misleading.  A single replication would always yield either 100% on time or 0% on time.  We wanted the answer to also give decision makers a sense of how confident they could be in the answer.  Using the Wilson Score, a single replication will yield a result of 60% at best and 40% at worst (using 95% confidence level).  This helps the decision maker identify that they have a very small sample of data and would encourage them to run additional replications. 

Can you give me an example of how this works?

Risk analysis can be demonstrated using any scheduling example.  It is best viewed in the Entity Gantt.  In the screenshots below, we’ve included 2 orders from the Candy Manufacturing Scheduling example.  One of the orders is overdue (will be late always), and the other has plenty of time (will be on time always).

The base rate is 50%.  After 1 replication, Simio updates the probabilities.  Order 1 now has a 60% on time probability.  Order 2 has a 40% on time probability.

After 2 replications, 67% and 33%:

After 5 replications, 78% and 22%:

After 100 replications, 98% and 2%:

Finally, after 1000 replications, 100% and 0%:

How many replications should I run?

By default, we suggest 10 replications (and 95% confidence level).  With these settings, a risk measure of 86% is a good sign, while 14% is a bad one.  Beyond the default settings, there are several additional factors which are dependent on the situation and use case.  One of these factors is slack time (the time between estimated completion and due date).  On the Gantt, slack time is the distance between the grey marker and the green marker.  If the slack time is large, a single replication may suffice.  If the slack time is small, additional replications will help identify if the order is in trouble or not. 

Now that I know my risk, what can I do about it?

Depending on your position in the organization (and therefore your decision rights), you can change either the design or operation of the system.  Example design changes include things like adding another assembly line or buying another forklift.  These changes are long term and may require approvals for capital expenditure (which the model facilitates by quantifying the impact of the expenditure).  Example operational changes include things like adding overtime, expediting a material, or changing order priorities, quantities, due dates etc.  Bridging the gap between design and operation are the dispatching rules, which relate to overall business objectives.  They are also flexible parameters which control how Simio chooses the next job from a queue (e.g., earliest due date, least setup, critical ratio, etc.).  All of these parameters influence risk and can be changed, provided that the user has the authority to change them.

Will Simio choose the best design and operation for me?

Decision rights and business processes have far reaching consequences.  A floor manager can probably authorize overtime if the schedule looks risky.  He probably cannot buy a piece of equipment.  To change a priority or a due date, he probably needs to consult with the commercial team and/or account managers.  To expedite a material, he probably needs to communicate with the procurement team.  To make a capital expenditure (i.e., change system design), he probably needs executive/financial approval.  Our solution respects those boundaries.  We treat priorities, due dates, etc. as inputs rather than outputs.  Any of these parameters can be changed by the appropriate decision maker.  They should not be changed by the tool without consent.  Simio assists the decision maker (at any level in the organization) by exposing the true consequences.

With so many choices, how can I quickly explore the consequences across multiple scenarios?

The experiment runner is used to explore consequences (which we call Responses) across multiple scenarios where a user can influence the parameters mentioned above (which we call Controls).  If the solution space is very large (i.e., there are many controls with a wide range of acceptable values), we recommend using OptQuest to automate the search of the solution space based on single or multiple objectives (e.g., low cost and high service level).  OptQuest uses a Tabu search which learns how the control values influence the objectives as it explores the solution space.

How often should I run these type of experiments?

Experiments are most relevant to design choices.  Operational decisions have many hard constraints which cannot be easily influenced.  For example, though Simio will allow you to adjust material receipt dates of critical materials and show you the impact on the schedule, many of them are inflexible and out of control of planner or even the business.  If you ask OptQuest how much inventory you would like to have, it will tell you, but this information adds no value because it is not actionable in the short term.  The planners need to work with what they have and make the best of it.  In practical application, we recommend running large experiments to explore design decisions on a monthly or quarterly basis.