Using movie reviews intelligently

We're in the thick of awards season, that febrile period of movie marketing madness when professional film criticism helps to gauge the year's best offerings, and not simply advise us which ticket to buy at the box office.

But as David A. Gross, Editor & Publisher of MovieReviewIntelligence.com, makes clear in the guest post below, we should be wary of aggregated results that smooth over subtle gradations in opinion. Methodology is critical.

There is a lot of commentary at the moment about which movies were well reviewed in 2010, which is the best reviewed movie, and the implications for Awards.

The Social Network and Toy Story 3 were both exceptionally well reviewed.  The Social Network's reviews are slightly better.  For example, Roger Ebert, Ann Hornaday and Christy Lemire each gave Toy Story 3 three stars, and Peter Travers gave it 3.5.  Each of these critics gave The Social Network four stars.

Some of the confusion comes from the methodology that Rotten Tomatoes uses to generate their score.  They classify each review as either positive or negative and then take the percentage of positive reviews.  That means that a B- review is the same as an A review, and a C- review is the same as an F.  They consider a C review or 2 out of 4 stars negative.  By not recognizing any gradation or anything in the middle, they push scores higher and lower.

How to Train Your Dragon was very well reviewed.  Critics gave it lots of Bs along with As.  But it is not in the uppermost echelon (its reviews were 77.2% positive; Rotten Tomatoes has it at 98%).

(An example of a movie in the middle is Eat Pray Love.  By pushing its middle scores down, it ended up with a 36% Tomato score.  According to the scores that the critics themselves chose, Eat Pray Love reviews were 56.2% positive.)

According to the critics themselves, which is what counts, the best reviewed movies of 2010 were

The Social Network (94.4% positive)
Toy Story 3 (92.1% positive)
The Kids are All Right (91.7% positive)

Rotten Tomatoes recently awarded best reviewed movie to Toy Story 3, while most critics groups have called The Social Network best movie.  The disconnect comes from the methodology flaw described above.

This is an issue the industry is currently focused on.  It would benefit from some clarification.

The following is a complete list of best reviewed movies in 2010, per Movie Review Intelligence.

David A. Gross
Editor & Publisher
MovieReviewIntelligence.com

Comments

[Soylent_grey]

Is much known if these aggregate scores feedback to the creatives and financial backers of films? I ask in part out of ignorance, but also as I know the games industry is increasingly growing to incorporate Metacritic scores as a device with which to calculate bonuses/funding for independent studios working under the auspices of larger publishers.

As you point out, the methodology if Rotten Tomatoes throws up all sorts of issues, I'm just curious if prestige critics matter more than aggregate scores, at least in the eyes of studios and financers. Or does it all just boil down to opening weekend box office anyways?

[Jim Barratt]

Good question, though I share your ignorance so I can't answer definitively!

My guess is that aggregate scores and quotes from prestige critics perform different functions. The former are employed by PRs when they want to claim a film was 'the best reviewed' of the year. The latter feature heavily in trade press ads and screener copy during the run up to the major awards. Quantitative and qualitative testimony, if you will.

(I didn't realise the games industry operated in the way you describe, but there's a certain logic to it).

Anyway, here's what David has to say:

'Interesting comment about the post. Reviews are one of many things that influence the perception of a movie and ultimately box-office performance. The industry follows reviews carefully. You will see a correlation between reviews and Awards, as they start to be announced over the next month or two.'