Cross-evaluation of metrics to estimate the significance of creative works
by dan_dassow | created - 30 Aug 2015 | updated - 09 Sep 2015 | PublicFull Article: http://www.pnas.org/content/112/5/1281.full
Supporting Information: http://www.pnas.org/content/suppl/2015/01/15/1412198112.DCSupplemental/pnas.201412198SI.pdf#nameddest=ST3
vol. 112 no. 5 > Max Wasserman, 1281–1286, doi: 10.1073/pnas.1412198112
Authors: Max Wasserman, Xiao Han T. Zeng, and Luís A. Nunes Amaral
Edited by Kenneth W. Wachter, University of California, Berkeley, CA, and approved December 1, 2014 (received for review June 27, 2014)
Significance
Whether it is Hollywood movies or research papers, identifying works of great significance is imperative in a modern society overflowing with information. Through analysis of a network constructed from citations between films as referenced in the Internet Movie Database, we obtain several automated metrics for significance. We find that the best automated method can identify significant films, represented by selection to the US National Film Registry, at least as well as the aggregate rating of many experts and far better than the rating of a single expert. We hypothesize that these results may hold for other creative works.
Abstract In a world overflowing with creative works, it is useful to be able to filter out the unimportant works so that the significant ones can be identified and thereby absorbed. An automated method could provide an objective approach for evaluating the significance of works on a universal scale. However, there have been few attempts at creating such a measure, and there are few “ground truths” for validating the effectiveness of potential metrics for significance. For movies, the US Library of Congress’s National Film Registry (NFR) contains American films that are “culturally, historically, or aesthetically significant” as chosen through a careful evaluation and deliberation process. By analyzing a network of citations between 15,425 United States-produced films procured from the Internet Movie Database (IMDb), we obtain several automated metrics for significance. The best of these metrics is able to indicate a film’s presence in the NFR at least as well or better than metrics based on aggregated expert opinions or large population surveys. Importantly, automated metrics can easily be applied to older films for which no other rating may be available. Our results may have implications for the evaluation of other creative works such as scientific research.
Key Words: data science complex networks citations films IMDb
List based on: Table S3. Films with most long-gap citations
LGC = *Long-gap citation count.
NFR year = Year inducted into the US Library of Congress’s National Film Registry (NFR)
- Instant Watch Options
- Genres
- Movies or TV
- IMDb Rating
- In Theaters
- Release Year
- Keywords
1. The Wizard of Oz (1939)
PG | 102 min | Adventure, Family, Fantasy
Young Dorothy Gale and her dog Toto are swept away by a tornado from their Kansas farm to the magical Land of Oz, and embark on a quest with three new friends to see the Wizard, who can return her to her home and fulfill the others' wishes.
Directors: Victor Fleming, King Vidor | Stars: Judy Garland, Frank Morgan, Ray Bolger, Bert Lahr
Votes: 428,882 | Gross: $2.08M
Year LGC NFR year 1939 565 1989
2. Star Wars: Episode IV - A New Hope (1977)
PG | 121 min | Action, Adventure, Fantasy
Luke Skywalker joins forces with a Jedi Knight, a cocky pilot, a Wookiee and two droids to save the galaxy from the Empire's world-destroying battle station, while also attempting to rescue Princess Leia from the mysterious Darth Vader.
Director: George Lucas | Stars: Mark Hamill, Harrison Ford, Carrie Fisher, Alec Guinness
Votes: 1,450,648 | Gross: $322.74M
Year LGC NFR year 1977 297 1989
3. Psycho (1960)
R | 109 min | Horror, Mystery, Thriller
A Phoenix secretary embezzles $40,000 from her employer's client, goes on the run and checks into a remote motel run by a young man under the domination of his mother.
Director: Alfred Hitchcock | Stars: Anthony Perkins, Janet Leigh, Vera Miles, John Gavin
Votes: 718,761 | Gross: $32.00M
Year LGC NFR year 1960 241 1992
4. Casablanca (1942)
PG | 102 min | Drama, Romance, War
A cynical expatriate American cafe owner struggles to decide whether or not to help his former lover and her fugitive husband escape the Nazis in French Morocco.
Director: Michael Curtiz | Stars: Humphrey Bogart, Ingrid Bergman, Paul Henreid, Claude Rains
Votes: 606,347 | Gross: $1.02M
Year LGC NFR year 1942 212 1989
5. Gone with the Wind (1939)
Passed | 238 min | Drama, Romance, War
A sheltered and manipulative Southern belle and a roguish profiteer face off in a turbulent romance as the society around them crumbles with the end of slavery and is rebuilt during the Civil War and Reconstruction periods.
Directors: Victor Fleming, George Cukor, Sam Wood | Stars: Clark Gable, Vivien Leigh, Thomas Mitchell, Barbara O'Neil
Votes: 334,482 | Gross: $198.68M
Year LGC NFR year 1939 198 1989
6. King Kong (1933)
Passed | 100 min | Adventure, Horror
A film crew goes to a tropical island for a location shoot, where they capture a colossal ape who takes a shine to their blonde starlet, and bring him back to New York City.
Directors: Merian C. Cooper, Ernest B. Schoedsack | Stars: Fay Wray, Robert Armstrong, Bruce Cabot, Frank Reicher
Votes: 91,019 | Gross: $10.00M
Year LGC NFR year 1933 191 1991
7. Frankenstein (1931)
Passed | 70 min | Drama, Horror, Sci-Fi
Dr Henry Frankenstein is obsessed with assembling a living being from parts of several exhumed corpses.
Director: James Whale | Stars: Colin Clive, Mae Clarke, Boris Karloff, John Boles
Votes: 79,757
Year LGC NFR year 1931 170 1991
8. The Godfather (1972)
R | 175 min | Crime, Drama
The aging patriarch of an organized crime dynasty transfers control of his clandestine empire to his reluctant son.
Director: Francis Ford Coppola | Stars: Marlon Brando, Al Pacino, James Caan, Diane Keaton
Votes: 2,014,097 | Gross: $134.97M
Year LGC NFR year 1972 162 1990
9. Citizen Kane (1941)
PG | 119 min | Drama, Mystery
Following the death of publishing tycoon Charles Foster Kane, reporters scramble to uncover the meaning of his final utterance: 'Rosebud.'
Director: Orson Welles | Stars: Orson Welles, Joseph Cotten, Dorothy Comingore, Agnes Moorehead
Votes: 466,180 | Gross: $1.59M
Year LGC NFR year 1941 143 1989
10. 2001: A Space Odyssey (1968)
G | 149 min | Adventure, Sci-Fi
After uncovering a mysterious artifact buried beneath the Lunar surface, a spacecraft is sent to Jupiter to find its origins: a spacecraft manned by two men and the supercomputer HAL 9000.
Director: Stanley Kubrick | Stars: Keir Dullea, Gary Lockwood, William Sylvester, Daniel Richter
Votes: 720,062 | Gross: $56.95M
Year LGC NFR year 1968 143 1991
11. Jaws (1975)
PG | 124 min | Adventure, Mystery, Thriller
When a killer shark unleashes chaos on a beach community off Cape Cod, it's up to a local sheriff, a marine biologist, and an old seafarer to hunt the beast down.
Director: Steven Spielberg | Stars: Roy Scheider, Robert Shaw, Richard Dreyfuss, Lorraine Gary
Votes: 659,936 | Gross: $260.00M
Year LGC NFR year 1975 129 2001
12. Night of the Living Dead (1968)
Not Rated | 96 min | Horror, Thriller
A ragtag group of Pennsylvanians barricade themselves in an old farmhouse to remain safe from a horde of flesh-eating ghouls that are ravaging the Northeast of the United States.
Director: George A. Romero | Stars: Duane Jones, Judith O'Dea, Karl Hardman, Marilyn Eastman
Votes: 139,226 | Gross: $0.09M
Year LGC NFR year 1968 122 1999
13. It's a Wonderful Life (1946)
PG | 130 min | Drama, Family, Fantasy
An angel is sent from Heaven to help a desperately frustrated businessman by showing him what life would have been like if he had never existed.
Director: Frank Capra | Stars: James Stewart, Donna Reed, Lionel Barrymore, Thomas Mitchell
Votes: 499,421
Year LGC NFR year 1946 109 1990
14. The Graduate (1967)
PG | 106 min | Comedy, Drama, Romance
A disillusioned college graduate finds himself torn between his older lover and her daughter.
Director: Mike Nichols | Stars: Dustin Hoffman, Anne Bancroft, Katharine Ross, William Daniels
Votes: 288,567 | Gross: $104.95M
Year LGC NFR year 1967 97 1996
15. Vertigo (1958)
PG | 128 min | Mystery, Romance, Thriller
A former San Francisco police detective juggles wrestling with his personal demons and becoming obsessed with the hauntingly beautiful woman he has been hired to trail, who may be deeply disturbed.
Director: Alfred Hitchcock | Stars: James Stewart, Kim Novak, Barbara Bel Geddes, Tom Helmore
Votes: 427,231 | Gross: $3.20M
Year LGC NFR year 1958 92 1989
16. Snow White and the Seven Dwarfs (1937)
Approved | 83 min | Animation, Adventure, Family
Exiled into the dangerous forest by her wicked stepmother, a princess is rescued by seven dwarf miners who make her part of their household.
Directors: William Cottrell, David Hand, Wilfred Jackson, Larry Morey, Perce Pearce, Ben Sharpsteen | Stars: Adriana Caselotti, Harry Stockwell, Lucille La Verne, Roy Atwell
Votes: 215,581 | Gross: $184.93M
Year LGC NFR year 1937 91 1989
17. Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1964)
PG | 95 min | Comedy, War
An unhinged American general orders a bombing attack on the Soviet Union, triggering a path to nuclear holocaust that a war room full of politicians and generals frantically tries to stop.
Director: Stanley Kubrick | Stars: Peter Sellers, George C. Scott, Sterling Hayden, Keenan Wynn
Votes: 518,550 | Gross: $0.28M
Year LGC NFR year 1964 91 1989
18. Dracula (1931)
Passed | 75 min | Drama, Fantasy, Horror
Transylvanian vampire Count Dracula bends a naive real estate agent to his will, then takes up residence at a London estate where he sleeps in his coffin by day and searches for potential victims by night.
Directors: Tod Browning, Karl Freund | Stars: Bela Lugosi, Helen Chandler, David Manners, Dwight Frye
Votes: 58,666
Year LGC NFR year 1931 90 2000
19. The Maltese Falcon (1941)
Passed | 100 min | Crime, Film-Noir, Mystery
San Francisco private detective Sam Spade takes on a case that involves him with three eccentric criminals, a gorgeous liar and their quest for a priceless statuette, with the stakes rising after his partner is murdered.
Director: John Huston | Stars: Humphrey Bogart, Mary Astor, Gladys George, Peter Lorre
Votes: 166,646 | Gross: $2.11M
Year LGC NFR year 1941 80 1989
20. Bambi (1942)
G | 69 min | Animation, Adventure, Drama
The story of a young deer growing up in the forest.
Directors: James Algar, Samuel Armstrong, David Hand, Graham Heid, Bill Roberts, Paul Satterfield, Norman Wright, Arthur Davis, Clyde Geronimi | Stars: Hardie Albright, Stan Alexander, Bobette Audrey, Peter Behn
Votes: 154,178 | Gross: $102.80M
Year LGC NFR year 1942 79 2011
21. The Exorcist (1973)
R | 122 min | Horror
When a young girl is possessed by a mysterious entity, her mother seeks the help of two Catholic priests to save her life.
Director: William Friedkin | Stars: Ellen Burstyn, Max von Sydow, Linda Blair, Lee J. Cobb
Votes: 455,293 | Gross: $232.91M
Year LGC NFR year 1973 78 2010
22. Taxi Driver (1976)
R | 114 min | Crime, Drama
A mentally unstable veteran works as a nighttime taxi driver in New York City, where the perceived decadence and sleaze fuels his urge for violent action.
Director: Martin Scorsese | Stars: Robert De Niro, Jodie Foster, Cybill Shepherd, Albert Brooks
Votes: 921,420 | Gross: $28.26M
Year LGC NFR year 1976 71 1994
23. Sunset Blvd. (1950)
Passed | 110 min | Drama, Film-Noir
A screenwriter develops a dangerous relationship with a faded film star determined to make a triumphant return.
Director: Billy Wilder | Stars: William Holden, Gloria Swanson, Erich von Stroheim, Nancy Olson
Votes: 236,581
Year LGC NFR year 1950 70 1989
24. Planet of the Apes (1968)
G | 112 min | Adventure, Sci-Fi
An astronaut crew crash-lands on a planet where highly intelligent non-human ape species are dominant and humans are enslaved.
Director: Franklin J. Schaffner | Stars: Charlton Heston, Roddy McDowall, Kim Hunter, Maurice Evans
Votes: 193,583 | Gross: $33.40M
Year LGC NFR year 1968 69 2001
25. Deliverance (1972)
R | 109 min | Adventure, Drama, Thriller
Intent on seeing the Cahulawassee River before it's dammed and turned into a lake, outdoor fanatic Lewis Medlock takes his friends on a canoeing trip they'll never forget into the dangerous American back-country.
Director: John Boorman | Stars: Jon Voight, Burt Reynolds, Ned Beatty, Ronny Cox
Votes: 119,340 | Gross: $7.06M
Year LGC NFR year 1972 66 2008
26. The Sound of Music (1965)
G | 172 min | Biography, Drama, Family
A young novice is sent by her convent in 1930s Austria to become a governess to the seven children of a widowed naval officer.
Director: Robert Wise | Stars: Julie Andrews, Christopher Plummer, Eleanor Parker, Richard Haydn
Votes: 261,246 | Gross: $163.21M
Year LGC NFR year 1965 61 2001
27. Bride of Frankenstein (1935)
Not Rated | 75 min | Drama, Horror, Sci-Fi
Mary Shelley reveals the main characters of her novel survived: Baron Henry Frankenstein, goaded by an even madder scientist, builds his monster a mate.
Director: James Whale | Stars: Boris Karloff, Elsa Lanchester, Colin Clive, Valerie Hobson
Votes: 53,291 | Gross: $4.36M
Year LGC NFR year 1935 58 1998
28. Singin' in the Rain (1952)
G | 103 min | Comedy, Musical, Romance
A silent film star falls for a chorus girl just as he and his delusionally jealous screen partner are trying to make the difficult transition to talking pictures in 1920s Hollywood.
Directors: Stanley Donen, Gene Kelly | Stars: Gene Kelly, Donald O'Connor, Debbie Reynolds, Jean Hagen
Votes: 260,834 | Gross: $8.82M
Year LGC NFR year 1952 57 1989
29. Apocalypse Now (1979)
R | 147 min | Drama, Mystery, War
A U.S. Army officer serving in Vietnam is tasked with assassinating a renegade Special Forces Colonel who sees himself as a god.
Director: Francis Ford Coppola | Stars: Martin Sheen, Marlon Brando, Robert Duvall, Frederic Forrest
Votes: 710,765 | Gross: $83.47M
Year LGC NFR year 1979 57 2000
30. The Texas Chain Saw Massacre (1974)
R | 83 min | Horror
Five friends head out to rural Texas to visit the grave of a grandfather. On the way they stumble across what appears to be a deserted house, only to discover something sinister within. Something armed with a chainsaw.
Director: Tobe Hooper | Stars: Marilyn Burns, Edwin Neal, Allen Danziger, Paul A. Partain
Votes: 183,805 | Gross: $30.86M
Year LGC NFR year 1974 57
31. Rebel Without a Cause (1955)
PG-13 | 111 min | Drama
A rebellious young man with a troubled past comes to a new town, finding friends and enemies.
Director: Nicholas Ray | Stars: James Dean, Natalie Wood, Sal Mineo, Jim Backus
Votes: 98,000
Year LGC NFR year 1955 57 1990
32. Star Wars: Episode V - The Empire Strikes Back (1980)
PG | 124 min | Action, Adventure, Fantasy
After the Rebel Alliance are overpowered by the Empire, Luke Skywalker begins his Jedi training with Yoda, while his friends are pursued across the galaxy by Darth Vader and bounty hunter Boba Fett.
Director: Irvin Kershner | Stars: Mark Hamill, Harrison Ford, Carrie Fisher, Billy Dee Williams
Votes: 1,380,744 | Gross: $290.48M
Year LGC NFR year 1980 56 2010
33. North by Northwest (1959)
Approved | 136 min | Action, Adventure, Mystery
A New York City advertising executive goes on the run after being mistaken for a government agent by a group of foreign spies, and falls for a woman whose loyalties he begins to doubt.
Director: Alfred Hitchcock | Stars: Cary Grant, Eva Marie Saint, James Mason, Jessie Royce Landis
Votes: 346,345 | Gross: $13.28M
Year LGC NFR year 1959 54 1995
34. Rear Window (1954)
PG | 112 min | Mystery, Thriller
A wheelchair-bound photographer spies on his neighbors from his Greenwich Village courtyard apartment window and, despite the skepticism of his fashion-model girlfriend, becomes convinced one of them has committed murder.
Director: Alfred Hitchcock | Stars: James Stewart, Grace Kelly, Wendell Corey, Thelma Ritter
Votes: 522,427 | Gross: $36.76M
Year LGC NFR year 1954 54 1997
35. Mary Poppins (1964)
G | 139 min | Comedy, Family, Fantasy
In turn of the century London, a magical nanny employs music and adventure to help two neglected children become closer to their father.
Director: Robert Stevenson | Stars: Julie Andrews, Dick Van Dyke, David Tomlinson, Glynis Johns
Votes: 186,146 | Gross: $102.27M
Year LGC NFR year 1964 54 2013
36. Pinocchio (1940)
G | 88 min | Animation, Adventure, Comedy
A living puppet, with the help of a cricket as his conscience, must prove himself worthy to become a real boy.
Directors: Norman Ferguson, T. Hee, Wilfred Jackson, Jack Kinney, Hamilton Luske, Bill Roberts, Ben Sharpsteen | Stars: Dickie Jones, Christian Rub, Mel Blanc, Don Brodie
Votes: 159,965 | Gross: $84.25M
Year LGC NFR year 1940 53 1994
37. Willy Wonka & the Chocolate Factory (1971)
G | 100 min | Adventure, Comedy, Family
A poor but hopeful boy seeks one of the five coveted golden tickets that will send him on a tour of Willy Wonka's mysterious chocolate factory.
Director: Mel Stuart | Stars: Gene Wilder, Jack Albertson, Peter Ostrum, Roy Kinnear
Votes: 227,519 | Gross: $4.00M
Year LGC NFR year 1971 52
38. The Seven Year Itch (1955)
Not Rated | 105 min | Comedy, Romance
When his family goes away for the summer, a hitherto faithful husband with an overactive imagination is tempted by a beautiful neighbor.
Director: Billy Wilder | Stars: Marilyn Monroe, Tom Ewell, Evelyn Keyes, Sonny Tufts
Votes: 42,502 | Gross: $12.00M
Year LGC NFR year 1955 51
39. Rosemary's Baby (1968)
Approved | 137 min | Drama, Horror
A young couple trying for a baby moves into an aging, ornate apartment building on Central Park West, where they find themselves surrounded by peculiar neighbors.
Director: Roman Polanski | Stars: Mia Farrow, John Cassavetes, Ruth Gordon, Sidney Blackmer
Votes: 235,354
Year LGC NFR year 1968 51
40. West Side Story (1961)
Approved | 153 min | Crime, Drama, Musical
Two youngsters from rival New York City gangs fall in love, but tensions between their respective friends build toward tragedy.
Directors: Jerome Robbins, Robert Wise | Stars: Natalie Wood, George Chakiris, Richard Beymer, Russ Tamblyn
Votes: 121,259 | Gross: $43.66M
Year LGC NFR year 1961 51 1997
41. Dirty Harry (1971)
R | 102 min | Action, Crime, Thriller
When a man calling himself "the Scorpio Killer" menaces San Francisco, tough-as-nails Police Inspector "Dirty" Harry Callahan is assigned to track down the crazed psychopath.
Directors: Don Siegel, Clint Eastwood | Stars: Clint Eastwood, Andrew Robinson, Harry Guardino, Reni Santoni
Votes: 168,409 | Gross: $35.90M
Year LGC NFR year 1971 51 2012
Tell Your Friends