Forecast accuracy metrics: the same forecast, scored four ways

One week had no orders and one week ran to 228. On the same twelve weeks MAE and RMSE point at different settings, MAPE returns no number at all, and is the only one of the four that says whether any of it beat the yardstick.

Live result

The forecast misses by 53.7 units a week, and by 78.9 once each miss is squared.

Weeks in the average3
Week with no orders9

A percentage needs something to divide by, and that week gives it nothing.

Try a series

The series with both forecasts on it

Horizontal: Week
Vertical: Units ordered a week
Units: Units a week for MAE and RMSE, per cent for MAPE, a bare ratio for MASE.

Against the yardstick set before the scoring began, this forecast reads 4.26. Above one means its average miss is larger than last week's number was missing by across the training weeks.

Fixed panels

MAE, lowest at 5 weeks.

RMSE, lowest at 6 weeks.

MAPE, no number at any setting.

MASE, lowest at 5 weeks, and the same curve as MAE.

Measure	3 week average	Last week
MAE, units a week	53.7	89.5
RMSE, units a week	78.9	120.5
MAPE	no number	no number
MASE	4.26	7.10
The MASE yardstick	12.6	12.6
Week with the largest error	week 10	week 10

Weeks in the average3

Week 9 orders

Week 10 orders228units

Which week each metric blames

Horizontal: Share of the metric's total, %
Basis: Each row is one metric's own total across the six scored weeks, set to 100.
Units: Per cent of the metric's own total, to one decimal.
Assumptions: Six scored weeks; shares taken from absolute error, squared error and percentage error in turn; the other four weeks pooled into one segment.

What a metric is choosing for you

Absolute and squared error differ in how much weight they give a large miss, and the relevant choice is how sharply the cost grows with that miss.

Percentage error divides by what happened, so a zero ends the calculation rather than creating a week to drop.

MASE compares absolute error with last week's number on the training weeks and always names the same setting as absolute error.

Here is the maths

Scale: The four fixed panels share one scale, weeks 1 to 6 across and 0 to 1.1 up, and it does not move. The live chart above rescales with what you type.
Basis: Weeks 1 to 6 fit nothing and hold the yardstick; weeks 7 to 12 are scored; the MASE yardstick is the naive error over weeks 2 to 6 alone.
Assumptions: One series, no seasonality; the split fixed before you touch anything; the yardstick computed on the training weeks only; every forecast a plain average of the weeks before it.
Model version: 1.0

The formula, in words

Split the weeks first, then score The first six weeks hold the yardstick and give the average enough history to start. The last six are the ones scored, and they stay fixed across every setting.
Four denominators, four questions Absolute error adds the misses up and divides by how many there were, in units. Squared error squares each one first, which lets the biggest week own more of the total, then takes a square root to get back to units. Percentage error divides each miss by what was actually ordered that week. Scaled error divides the whole absolute error by one number, the average week to week change across the training weeks.
The yardstick is frozen, and that is the point of the fourth one The number MASE divides by, 12.6 units a week here, is computed on the training weeks alone and never on the weeks being scored. Recompute it from the scored weeks and you have used information the forecast never had, and the ratio stops meaning anything. Because it is a constant, MASE and absolute error are the same shape: it makes the number comparable across series, not independent of the other three.
A metric ranks weeks as well as methods Divide each week's contribution by the metric's own total and you get how much of the score that week owns. Squaring hands more of it to the biggest miss. Dividing by what was ordered hands more of it to the smallest week. That is why the week you would go and ask about changes with the metric, on error columns that are identical.

e(t)=A(t)-F(t)

where

A(t)

is the actual and

F(t)

the forecast.

Absolute error:

\mathrm{MAE}=\frac{\sum|e|}{6}

Squared error:

\mathrm{RMSE}=\sqrt{\frac{\sum e^2}{6}}

Percentage error:

A(t)=0\implies\mathrm{MAPE}=\bot

where

\bot

is no value.

Scaled error:

\mathrm{MASE}=\frac{\mathrm{MAE}}{12.6}

d

is the MASE denominator.

A worked example

Shipped defaults: three weeks in the average, week 9 at no orders, week 10 at 228.

The twelve weeks. Training, weeks 1 to 6: 120, 108, 132, 117, 126, 123. Scored, weeks 7 to 12: 117, 129, 0, 228, 96, 126.

The yardstick, fixed before anything is scored. Take the five week to week changes inside the training window only: $\left|108 - 120\right| = 12$ , $\left|132 - 108\right| = 24$ , $\left|117 - 132\right| = 15$ , $\left|126 - 117\right| = 9$ , $\left|123 - 126\right| = 3$ . Their total is 63, and $\frac{63}{5} = 12.6$ units a week $d=\frac{12+24+15+9+3}{5}=12.6$ .

The six forecasts and the six errors.

Week	The three weeks averaged	Forecast	Actual	Error	Squared	Percentage
7	117, 126, 123	122	117	−5	25	4.3%
8	126, 123, 117	122	129	+7	49	5.4%
9	123, 117, 129	123	0	−123	15,129	no number
10	117, 129, 0	82	228	+146	21,316	64.0%
11	129, 0, 228	119	96	−23	529	24.0%
12	0, 228, 96	108	126	+18	324	14.3%
				322	37,372	no number

The four scores. $\mathrm{MAE} = \frac{322}{6} \approx 53.7$ units a week. The mean squared error is $\frac{37{,}372}{6}$ . $\mathrm{RMSE} = \sqrt{\frac{37{,}372}{6}} \approx \sqrt{6{,}228.67} \approx 78.9$ units a week. MAPE has no value because week 9 would divide by zero. $\mathrm{MASE} = \frac{53.7}{12.6} \approx 4.26$ .

The same six weeks under last week's number. Its forecasts are 123, 117, 129, 0, 228, 96, so its errors are −6, +12, −129, +228, −132, +30. Those total 537, giving $\mathrm{MAE}=89.5$ , and their squares total 87,129, giving $\mathrm{RMSE} = \sqrt{14{,}521.5} \approx 120.5$ . Its MASE is $\mathrm{MASE}=\frac{89.5}{12.6}\approx7.10$ .

Where they stop agreeing. Week 10 owns $\frac{21{,}316}{37{,}372} \approx 57.0\%$ of squared error but only $\frac{146}{322} \approx 45.3\%$ of absolute error, a difference of 11.7 points. Week 9 owns 40.5% of squared error against 38.2% of absolute error.

Slide the weeks in the average from 5 to 6 and MAE rises from 49.5 to 49.6 while RMSE falls from 73.4 to 73.1. MAE names five weeks and RMSE names six, on the same series and at the same moment.

Switch week 9 back to 114 units. The forecasts from week 10 on become 120, 157 and 146, so the errors become −5, +7, −9, +108, −61, −20. MAE is 35.0, RMSE is 51.5 and MAPE is 24.1%. Absolute and squared error name week 10, while percentage error names week 11.

Check it yourself

Set week 9 back to 114 units and read the two worst weeks off Chart 2. Absolute error and squared error both point at week 10, where the line ran to 228 against a forecast of 120. Percentage error points at week 11 instead, where a smaller miss landed on the smallest order of the six.

What this does not model

Seasonality and the seasonal naive denominator MASE uses on a seasonal series, rolling origins and any second forecast origin, prediction intervals and their coverage, any method beyond a plain average of the weeks before, bias as something separate from the size of a miss, what a miss costs in either direction, weighted or asymmetric loss, the intermittent demand methods built for series that are mostly zeros, and the fact that a setting chosen on six scored weeks has been chosen on very little.

The four definitions and their accompanying cautions come from Rob J. Hyndman and George Athanasopoulos, Forecasting Principles and Practice, and independently Nahmias and Olsen, Production and Operations Analysis. Both are described here and no text is reproduced. The twelve weeks are a series written for this page, not anyone's orders. Every number is arithmetic on the values above.