Zillow spends $1M trying to improve its ‘Zestimates’
SEATTLE – Feb. 4, 2019 – Zillow's estimate of a home's value, called the Zestimate, can be powerful: Some homeowners track them like a stock, and when it gets to a certain point, they may decide to sell. Home shoppers gauge the estimate against the list price of a home. Others use it just to gawk at their neighbor's home values.
But it's far from perfect: In Seattle, the Zestimate is off by a median of 4.7 percent compared to the actual sale price, according to the company – a $35,000 difference on the typical house. Real-estate brokers have long complained that the numbers give sellers, in particular, a distorted view of their home's true worth.
Now the Zestimate, that little number that appears at the top of every home's Zillow page and updates daily, is in line to get more accurate.
On Wednesday, the Seattle-based company awarded a $1 million prize to the winners of a public contest to improve its algorithm. The winning team, three guys from Raleigh, Toronto and Morocco who teamed up despite never having met in person, came up with a way to beat Zillow's own data scientists to a better estimate.
The contest started a year and a half ago with 3,800 teams from 91 countries and was narrowed down to 100 finalists last year. The teams were given seven years' worth of data on a sample of millions of homes across the country, and were tested to see how closely their estimated values for each home matched up with the actual sale prices of homes that sold in the ensuing months.
Jordan Meyer, the American on the winning team, reduced his workload at his day job as CTO of an analytics company and poured about six hours a day into the contest, communicating with his teammates, Moroccan computer science professor Chahhou Mohamed and Canadian artificial intelligence startup founder Nima Shahbazi, on the messaging application Slack.
Meyer started by finding every data source he could – the exact longitude and latitude of houses could be used to determine the proximity to streets and therefore determine noise near the house. Slight differences in distance from a body of water could influence a home price by thousands of dollars. In the end each home had hundreds of different data points.
But the strategy that set them apart was trying wildly different algorithms and merging the ones that worked together to get the best blended average.
"It was extremely hard," Meyer said in an interview. He called the process "relentless experimentation" and echoed Shahbazi, who said in a statement: "For every idea that worked, there were a hundred that didn't work. But we kept going."
Zillow has slowly improved its Zestimate from a median error rate of 14 percent when it started in 2006 to 5.7 percent when the contest began in mid-2017. It's now down to 4.5 percent nationally (it's higher in some cities and lower in others), and once the winners' tweaks to the algorithm are incorporated, the company expects the error rate to dip to about 4 percent.
"We're happy with the progress we're making." said Stan Humphries, Zillow's chief analytics officer. "You're going to get some way off. We do 115 million of these every day," referring to the number of homes on Zillow with a Zestimate, "so yes, we get concerned when we're off, and we're committed to making them even more accurate. This is an important number. The implications of getting it right are really important."
Humans are still better than machines
Homes nationally sell on average for about 2 percent less than the list price set by brokers, according to data from Redfin. Brokers have access to information that an algorithm often doesn't – the Zestimate relies on publicly available data and voluntary input from homeowners, which can give an incomplete picture of a house.
"There are way too many factors for a certain algorithm to work," said Sam Mansour, a managing broker with John L. Scott in Lynnwood. He said he constantly has to battle with clients who cling to their Zestimate. "I've been to homes and people say 'my Zestimate is worth X amount,' and I'm like, 'no, no.'"
He said he's also heard of homeowners who use their Zestimate, and the company's one-year forecast of their home value, to justify how much they'd like to borrow against their home. (Zestimates aren't used in official proceedings, like a home appraisal or a home-equity loan.)
Mansour said the biggest factor a computer can't track is the emotional appeal of a home, which can vary wildly from buyer to buyer and is a primary driver in how much people offer. And certain attributes that get plugged into algorithms are going to be weighed differently by various buyers – a large lot might appeal to some, but to others, it just means extra yard work.
Sometimes Zillow is really off – the median error rate of 4.5 percent nationally means half of home values are wrong by more than 4.5 percent.
Zillow says about 1 in 8 Zestimates winds up being wrong by at least 20 percent. That includes the 2016 home sale made by Zillow CEO Spencer Rascoff – who sold his Seattle home for 40 percent less than his Zestimate. In some counties where public data isn't great or there aren't many homes, Zestimates don't exist, or the median error rate can be above 10 percent.
Zillow is the most-clicked real estate site in the nation and was the first to offer a home-value estimator, but these days other websites like Redfin and realtor.com also offer their own home-value estimates.
The winners of the prize agreed to split their $1 million share evenly. As for what Meyer will do with his cut?
"I'll be investing in real estate for sure," he said.
© 2019 The Seattle Times, Mike Rosenberg. Distributed by Tribune Content Agency, LLC.