Netflix Prize Update: New Team in First Place (deadline tomorrow)(netflixprize.com)
netflixprize.com
Netflix Prize Update: New Team in First Place (deadline tomorrow)
http://www.netflixprize.com//leaderboard?limit=50
25 comments
The Ensemble is actually a meta-team of most of the teams that didn't participate in BPC, and I'm hearing that BPC had lately tried to get some other teams to join, so they were kinda expecting that (but I don't think they will be able to fire back).
Bottom line, I think that the most massive open data set out there (next to a competition) has really helped the research to move on, but I don't see the point in 5 or 10 teams joining forces to improve the results for a tiny percentage and win the prize.
On the other hand, maybe this is actually the right thing, for all these teams that have struggled for two years to be in the top-15 to receive the prize...
> I don't see the point in 5 or 10 teams joining forces to improve the results for a tiny percentage and win the prize.
I think the point is who gets the million dollars. It makes sense that people would group up, probably into as few additional participants as it would take to ensure winning. If you and I were in the vicinity of being able to win independently and we knew we would lose otherwise, I think we have every incentive to team up.
I think the point is who gets the million dollars. It makes sense that people would group up, probably into as few additional participants as it would take to ensure winning. If you and I were in the vicinity of being able to win independently and we knew we would lose otherwise, I think we have every incentive to team up.
BPC is a meta-team as well. It's made up of four other teams. Some members of BPC won the Progress Prizes in previous years.
I can completely understand joining forces to win the prize. $1,000,000 isn't chump change, and a smaller percentage of it is better than none.
I can completely understand joining forces to win the prize. $1,000,000 isn't chump change, and a smaller percentage of it is better than none.
Shouldn't a new result above 10 pct reset the clock with a new one month period ?
From the rules: http://www.netflixprize.com/rules
When the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging. At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.
When the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging. At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset.
Maybe it should, maybe it shouldn't, but Netflix defined it as 30 days from the first submission over a 10% improvement. I'm sure there's some reason why they did it that way, but I don't know what it is.
One thing's for sure: BellKor's Pragmatic Chaos is going to get shitfaced tonight.
Trying to hit the Ballmer Peak.
BK/PC submitted their last RMSE 2009-07-25 15:53:34, three hours before The Ensemble. If they can hit the Ballmer Peak or otherwise beat the Ensemble RMSE before tomorrow morning, it should be very interesting.
Anyone know why blending seems to work so well?
I believe it is a consequence of A) netflix evaluating the algorithms in terms of the squared error and B) different classification algorithms making different types of mistakes.
Take a simple example, with two movies (A and B), and two algorithms (X and Y). Say Alg. X rates movie A perfectly, but is off by 2 for movie B. Meanwhile, Alg. Y rates movie B perfectly, but is off by 2 for movie A.
The squared errors for the different algorithms will be:
Alg X: 4 (=0^2 + 2^2)
Alg Y: 4 (=2^2 + 0^2)
Mean of Alg X and Alg Y: 2 (=1^2 + 1^2)
Take a simple example, with two movies (A and B), and two algorithms (X and Y). Say Alg. X rates movie A perfectly, but is off by 2 for movie B. Meanwhile, Alg. Y rates movie B perfectly, but is off by 2 for movie A.
The squared errors for the different algorithms will be:
Alg X: 4 (=0^2 + 2^2)
Alg Y: 4 (=2^2 + 0^2)
Mean of Alg X and Alg Y: 2 (=1^2 + 1^2)
It's a standard ml technique that's been shown to work well, although no ones shown a proof of this. This is related to the observation "more data beats better models." This does seem to have it's limits - at some point you do need better features before you can expect your data to pay off.
Only because each individual algorithm is slightly overfit. Blending it all together keeps the good predictive part and reduces the overfit noise.
It seems like just a consequence of basic statistics (central limit theorem?). Any algorithm you create has some random bias. By blending, you take several algorithms with random bias and average them together, reducing the overall bias (assuming the algorithms converge toward the correct result).
That's as far from the central limit theorem as you can go..
Wow, the second team will be gutted!
If BPC didnt expect this to happen they deserve to lose. They should have expected a last minute submission and have one for themselves also.
Reducing the RMSE isn't quite that simple. Teams have been working for years to improve the algorithm by 10%. You can't just "boost it up" in order to win. It takes serious work. BPC is still working to improve their algorithm like everyone else but questioning whether they "expected this to happen" is ridiculous. None of the top teams "deserve to lose", they've all done great work and deserve a round of applause, regardless if they are the ultimate winner.
Yeah, the timestamp right there next to BPC is 07-25
[deleted]
[deleted]
Is there any rolling-extension each time a new top submission arrives -- like in some auctions -- or is the deadline firm at X days after the first +10% entry?
no. it's over tomorrow morning at ~11:30am PST... there are a few complications that could cause an extension, but they're not likely.