Monte Carlo simulation: one run is vivid and still an anecdote

Basis

Two stage delay chain

Distribution

stage 1 {1,2,3,5}, stage 2 {2,3,4,7}, equal weights

Seed

4021

Replications

1 replication

Model version

1

The formula, in words

• One run is an observation, not an answer. Each run draws one value for each stage and adds them, producing one of ten possible finishes. A single run carries no information about how far the other runs would have spread, so no estimate and no interval can be built from it.
• The mean estimates, and the half width says how precisely. Average the run totals for the sample mean; divide the sum of squared deviations by the run count less one and take the root for the sample standard deviation; divide that by the square root of the run count for the standard error; multiply by the t multiplier at 95 per cent for the half width.
• Four times the runs buys half the interval. The standard error falls with the square root of the run count rather than with the count, so the half width falls from 1.02 to 0.48 to 0.24 days across 25, 100 and 400 runs. Sixteen times the computation for a quarter of the interval, and that ratio does not improve by trying harder.
• The spread is not the interval. The standard deviation describes how much consignments differ from each other and does not shrink as runs accumulate. The interval describes how precisely the mean has been pinned down and does nothing else. The first is a fact about the world, the second is a fact about your effort.

• independent, one stream per stage
• each run starts empty
• both stages complete

A worked example

The experiment, by hand. Stage 1 averages $\frac{1+2+3+5}{4}=2.75$ days. Stage 2 averages $\frac{2+3+4+7}{4}=4.00$ days. The finish averages 6.75 days, and because the draws are independent the variances add: $2.1875 + 3.5 = 5.6875$, so the true standard deviation is $\sqrt{5.6875} = 2.3848$ days. Both are exact. Nothing was simulated to get them.

The one run. Seed 4021, run 1. Stream A emits 0.090384; $\lfloor0.090384\times4\rfloor=0$, so stage 1 takes 1 day. Stream B emits 0.315320; $\lfloor0.315320\times4\rfloor=1$, so stage 2 takes 3 days. The chain finishes in 4 days.

Four days is 2.75 days below the true mean, which is 41 per cent low and 1.15 true standard deviations out. It is not a freak: across a thousand runs, 192 finish in 4 days or fewer. A run that ordinary, watched once, is wrong by two fifths.

The interval at three replication counts, every step shown.

The law, watched twice. Twenty five runs to one hundred is four times the work, and the half width falls from 1.02 to 0.48, a factor of 2.11. One hundred to four hundred is four times the work again, and it falls from 0.48 to 0.24, a factor of 1.99. Four times the runs buys half the interval, and nothing buys better than that by replication alone. Going from twenty five runs to four hundred is sixteen times the computation for a quarter of the interval, and it moved the sample mean from 5.96 to 6.71, which is to say the estimate was never the thing that was improving quickly.

Where the vivid run was the whole time. At every one of the six replication counts, the interval contains the true mean of 6.75. At every one of them, the run the visitor watched sits at 4 days, outside every interval from $n=25$ onward. The run was never wrong and it was never evidence. Those are different failures and only one of them is fixable by watching more carefully.

The seed does not move the answer. Seed 31337 draws 0.704391 and 0.929328 on its first run, giving 3 days and 7 days for a finish of 10 days, which is 48 per cent above the true mean where seed 4021's was 41 per cent below. The two vivid runs are 4 days and 10 days, a spread of two and a half times, from the identical model. At a thousand runs the two seeds give means of 6.67 and 6.74, intervals of 6.52 to 6.81 and 6.59 to 6.89, and both contain 6.75. Change the seed and the story changes completely while the answer does not move at all.

Check it yourself

type 31337 into the seed and leave the replications at one. The first run finishes in 10 days instead of 4, from an experiment in which nothing changed. Now take both seeds to 1,000 runs: 4021 gives a mean of 6.67 and 31337 gives 6.74, and both intervals contain 6.75.

What this does not model

dependence between the two stages, which is a model component rather than a footnote and which this model declares as independent because it is; a common shock delaying both stages together, which independent sampling understates; any uncertainty about the model's structure or about the four outcomes in each set, which the sample variance measures nothing about and which no number of runs would reduce; variance reduction of any kind, including antithetic inputs, control variates, conditioning and stratification, each of which buys precision more cheaply than replication and none of which appears here; common random numbers and the paired comparison of two policies, which is the natural next page and which can increase variance rather than reduce it when matching goes the wrong way; warmup, steady state and any non terminating design, this experiment terminating when both stages finish; continuous distributions, fitted families and the question of which family the data supports; and any real shipment, route, operation or cost, none of which this experiment describes.