The Forecast Nobody Tested

In November 2021, Zillow's board decided to wind down Zillow Offers, the business that bought homes with the company's own money and resold them. Zillow wrote down about $304 million of inventory in the third quarter alone, warned of more losses on homes still in the pipeline, and said it would cut roughly a quarter of its workforce.

CEO Rich Barton put the cause plainly: the unpredictability of forecasting home prices was far greater than the company had anticipated.

The whole business ran on a forecast of what a house would be worth when it was resold. The forecast was the sign. The price a buyer would actually pay was the thing. Zillow reported the write-down as reflecting homes it had bought above their estimated resale value, and a sign that loose, held in inventory and paid for with real money, becomes a loss.

Here is the meta-learning point. A company can correct a forecast in many ways: more data, a retuned model, a better feature. Those are fixes inside the existing bet. The deeper question sits upstream: should the company be paying for houses on the strength of a forecast it cannot test until after the money has moved?

Most AI initiatives are funded on a confident prediction. Very few are funded with a plan to find out early that the prediction was wrong.

What forecast does your next big decision depend on, and how would you learn early that it was wrong?