What happened at Zillow?
Maybe, with your day-jobs, you haven’t been paying attention to how this has played out, but I think there are lessons hear to be learned. Given I now have a bit of time on my hand, I thought I would do a deep-dive into the publicly available information.
This afternoon, we announced financial results for the third quarter and, most notably, our decision to wind down our Zillow Offers operations, which will unfortunately involve a reduction in our workforce of approximately 25% over the next few quarters.
… ultimately, we determined that further scaling up Zillow Offers is too risky, too volatile to our earnings and operations, too low of a return on equity opportunity and too narrow in its ability to serve our customers, a tough but necessary determination.(Rich Barton, Zillow CEO, Q3 Earnings call)
First, some background.
We all know Zillow; most of us think of it as that website you visit to see an estimate of the value of your house. That estimate is based on algorithms that Zillow has developed over the last decade. Zillow calls its estimate a Zestimate.
There are numerous models that underlie their prediction, using factors such as home characteristics including square footage, location, the number of bathrooms, the distance to a highway, the distance to water, etc., on-market data such as listing price, description, comparable homes in the area and days on the market, off-market data — tax assessments, prior sales and other publicly available records and market trends, including seasonal changes in demand.1 The algorithm has been upgraded with new image-recognition technology that can pore over the photos included in for-sale listings and identify key features — like kitchen countertops, fireplaces and bathroom fixtures — that may play into the sale price2. The new approach apparently even includes the use of “deep learning.”
There is an overlay model that varies the weights given to the different feature models depending on location.
we model the appreciation for individual properties as:
To specify the function f(PropChar) we first define:
Then the function f(PropChar) is specified as follows:
f(PropChar)=α+β*HedonicCharacteristics+γ*ZestRelApprect
(graphic from Data Science At Zillow (Zillow 2/25/2015))
The algorithm was characterized as using AI from the beginning3. For example:
The Zestimate is generated through a series of processes built using various tools, including heavy doses of R, Python, Pandas, Scikit Learn, and GraphLab Create, the graph analytics software developed by Seattle-based Dato (formerly GraphLab)
Zillow uses a gradient-boosted random forest to match features on known fraudulent listings against new listings. The output from the machine learning algorithm is scored as actual fraud or not, and added back into the fraud model every week.
Zillow’s coverage increased significantly from 2006 to 2021, and the estimated accuracy of its algorithms improved materially as well.
In 2006, Mullaney stated Zillow had informed BusinessWeek that it would be able to obtain a value estimate within ±10% of actual value for 62% of homes4. Fifty-nine percent of the Zillow estimates fall within ±10% of the sale price and only 0.88% of values are underestimated by more than 10%.
In 2011, Zillow launched their 3rd update to the algorithm. “For the three-month period ending March 31, 2011, our national median error was 8.5 percent. The median error of our previous algorithm over the same time period was 12.7 percent, meaning that the new Zestimates are 33 percent more accurate than the older ones.5”
A 2016 update “improve[d] the national median error rate from 8 percent to 6 percent.6”
Finally, in 2019 “For homes listed for sale, the error rate is now less than 2%, meaning half of all Zestimates fall within 2% of the home's eventual sale price.7”
Looking back, Zillow’s approach to the Zestimate clearly changed XXX. Originally, Zillow stated:
First, as we expected, real estate professionals who know the local market and intimate details of the home can price a home better than an automated valuation model. But the Zestimate does provide a good starting point for a conversation about home values, a starting point that compares fairly well to list price, particularly when considering only listings that can be definitively ruled out as being underpriced. But, as we’ve always noted, that pricing conversation that may be started with the review of the Zestimate should ultimately be augmented with the input of opinions from local real estate professionals (agents, brokers and appraisers)8.
Zillow started as an advertising-based business model. In 2Q’18, Zillow launched Zillow Offers.
If a homeowner accepts an offer from Zillow Offers, Zillow buys the house, makes certain repairs and updates, and then lists it for sale on the open market.
Management immediately saw high demand and quickly grew the business. In their 2018 annual report, Zillow stated its ambition:
Purchase 5,000 homes per month through Zillow Offers, generating annualized revenue of approximately $20 billion, up from 686 homes purchased in 2018, which generated $52 million in revenue.
Justin Patterson from Raymond James got to the heart of the matter on the 4Q’18 earnings call:
I wanted to tease out your commentary on Zestimates being a key advantage in the Offers space. Historically, as we look at Zillow there tends to be a large gap between your Zestimates and where less prices are market-by-market.
The CFO responded:
you'll see us in every market price based on the Zestimates and based on a local market opinion, based on a pricing underwriting model and then ultimately based on visiting the home with the seller.
I mean approaching 4% median absolute percent error, it's the most accurate AVM (ph) out there in the country -- across the board and it is a fantastic tool for us to price off, and first thing we can start off.
Zillow appears to have styled themselves as a market maker:
When we decided to take a big swing on Zillow Offers 3.5 years ago, our aim was to become a market maker, not a market risk taker. And this was underpinned by the need to forecast the price of homes accurately three to six months into the future.9
So what went wrong?
We have been unable to accurately forecast future home prices at different times in both directions by much more than we modeled as possible, with Zillow Offers unit economics on a quarterly basis swinging from plus 576 basis points in Q2 to an expected minus 500 to minus 700 basis points in Q4.
The first clear issue is that they didn’t properly account for adverse selection; this is a text-book asymmetrical information problem. As far back as 2012 and up until 2017 they were humble about their models performance:
[R]eal estate professionals who know the local market and intimate details of the home can price a home better than an automated valuation model.
Second, from the above statements, it is also possible that the accuracy of its models may have deteriorated during their startup period during covid. Clearly the training data would not include the dynamics of what has occurred over the last two years, and it seems clear with hindsight that ramping up this business during this abnormal period was a management mistake.
Third, they really weren’t a market maker in the true sense. There is no evidence that they sought to hedge their inventory. They weren’t bringing together buyers and sellers; they were taking principal risk.
Clearly they were more like the folks on ‘flip-this-house’ writ large. A major part of their Zillow Offers business was to invest in and fix up houses prior to sale. And it appears that this effort was hampered during the covid epidemic:
As Bloomberg first reported, Zillow says it has developed a backlog of properties that it owns and needs to repair, inspect, and get back on the market. The employees who work in those roles—contractors, inspectors, and agents—are stretched too thin, so Zillow had to stop buying additional homes until it could deal with the ones it already owns.
As a result, their inventory expanded beyond what their balance sheet could handle. Zillow had purchased over 15,000 houses.
There is also a question about incentives. Zillow DID employ staff to visit each property to finalize the offers. From internet reporting (take with a grain of salt), many of the final offers were revised UP from the Zestimated value. How frequently this occurred is not obvious. But given growth expectations, it seems worth considering what incentives were in place for those making the final offers. If their compensation was based on aggregate sales, one can easily see perverse incentives.
What is a Zestimate? (Zillow 2021)
Zillow uses AI to hone its Zestimate tool (7/1/2019)
Zillow Unveils Smarter, More Accurate Zestimate That 'Sees' Unique Home Features, Incorporates Greater Real-Time Data ((Zillow 6/27/2019)
The new Zestimate algorithm leverages neural networks, the latest machine learning approach, and incorporates deeper history of property data such as sales transactions, tax assessments and public records, in addition to home details such as square footage and location.
Q&A: Zillow's Rich Barton on Real Estate, AI and Basement Floods (Wired 11/27/07)
Our algorithm is a complex piece of AI that pores through a ton of data, looks for patterns, and creates predictive models. Then it goes through by zip code and identifies which models work best in each neighborhood. We're going to take that all the way down to the house level eventually. It's an interesting computer science and stats problem.
Zillow’s Estimates of Single-Family Housing Values (The Appraisal Journal, Winter 2010)
Upgrading the Zestimate (June 2011)
Putting Accuracy in Context (Zillow 9/21/12)