The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations

The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations

Paperback(Reprint)

$15.26 $16.95 Save 10% Current price is $15.26, Original price is $16.95. You Save 10%.
View All Available Formats & Editions
Eligible for FREE SHIPPING
  • Want it by Wednesday, September 26?   Order by 12:00 PM Eastern and choose Expedited Shipping at checkout.
    Same Day shipping in Manhattan. 
    See Details

Overview

The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations by James Surowiecki, James Surowiecki

In this fascinating book, New Yorker business columnist James Surowiecki explores a deceptively simple idea: Large groups of people are smarter than an elite few, no matter how brilliant—better at solving problems, fostering innovation, coming to wise decisions, even predicting the future.

 

With boundless erudition and in delightfully clear prose, Surowiecki ranges across fields as diverse as popular culture, psychology, ant biology, behavioral economics, artificial intelligence, military history, and politics to show how this simple idea offers important lessons for how we live our lives, select our leaders, run our companies, and think about our world.

Product Details

ISBN-13: 9780385721707
Publisher: Knopf Doubleday Publishing Group
Publication date: 08/16/2005
Edition description: Reprint
Pages: 336
Sales rank: 123,157
Product dimensions: 5.20(w) x 7.94(h) x 0.74(d)

About the Author

James Surowiecki is a staff writer at The New Yorker, where he writes the popular business column, “The Financial Page.” His work has appeared in a wide range of publications, including the New York Times, the Wall Street Journal, Artforum, Wired, and Slate. He lives in Brooklyn, New York.

 

www.wisdomofcrowds.com

Read an Excerpt

The Wisdom of Crowds


I


If, years hence, people remember anything about the TV game show Who Wants to Be a Millionaire?, they will probably remember the contestants' panicked phone calls to friends and relatives. Or they may have a faint memory of that short-lived moment when Regis Philbin became a fashion icon for his willingness to wear a dark blue tie with a dark blue shirt. What people probably won't remember is that every week Who Wants to Be a Millionaire? pitted group intelligence against individual intelligence, and that every week, group intelligence won.

Who Wants to Be a Millionaire? was a simple show in terms of structure: a contestant was asked multiple-choice questions, which got successively more difficult, and if she answered fifteen questions in a row correctly, she walked away with $1 million. The show's gimmick was that if a contestant got stumped by a question, she could pursue three avenues of assistance. First, she could have two of the four multiple-choice answers removed (so she'd have at least a fifty-fifty shot at the right response). Second, she could place a call to a friend or relative, a person whom, before the show, she had singled out as one of the smartest people she knew, and ask him or her for the answer. And third, she could poll the studio audience, which would immediately cast its votes by computer. Everything we think we know about intelligence suggests that the smart individual would offer the most help. And, in fact, the "experts" did okay, offering the right answer--under pressure--almost 65 percent of the time. But they paled in comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.

Now, the results of Who Wants to Be a Millionaire? would never stand up to scientific scrutiny. We don't know how smart the experts were, so we don't know how impressive outperforming them was. And since the experts and the audiences didn't always answer the same questions, it's possible, though not likely, that the audiences were asked easier questions. Even so, it's hard to resist the thought that the success of the Millionaire audience was a modern example of the same phenomenon that Francis Galton caught a glimpse of a century ago.

As it happens, the possibilities of group intelligence, at least when it came to judging questions of fact, were demonstrated by a host of experiments conducted by American sociologists and psychologists between 1920 and the mid-1950s, the heyday of research into group dynamics. Although in general, as we'll see, the bigger the crowd the better, the groups in most of these early

experiments--which for some reason remained relatively unknown outside of academia--were relatively small. Yet they nonetheless performed very well. The Columbia sociologist Hazel Knight kicked things off with a series of studies in the early 1920s, the first of which had the virtue of simplicity. In that study Knight asked the students in her class to estimate the room's temperature, and then took a simple average of the estimates. The group guessed 72.4 degrees, while the actual temperature was 72 degrees. This was not, to be sure, the most auspicious beginning, since classroom temperatures are so stable that it's hard to imagine a class's estimate being too far off base. But in the years that followed, far more convincing evidence emerged, as students and soldiers across America were subjected to a barrage of puzzles, intelligence tests, and word games. The sociologist Kate H. Gordon asked two hundred students to rank items by weight, and found that the group's "estimate" was 94 percent accurate, which was better than all but five of the individual guesses. In another experiment students were asked to look at ten piles of buckshot--each a slightly different size than the rest--that had been glued to a piece of white cardboard, and rank them by size. This time, the group's guess was 94.5 percent accurate. A classic demonstration of group intelligence is the jelly-beans-in-the-jar experiment, in which invariably the group's estimate is superior to the vast majority of the individual guesses. When finance professor Jack Treynor ran the experiment in his class with a jar that held 850 beans, the group estimate was 871. Only one of the fifty-six people in the class made a better guess.

There are two lessons to draw from these experiments. First, in most of them the members of the group were not talking to each other or working on a problem together. They were making individual guesses, which were aggregated and then averaged. This is exactly what Galton did, and it is likely to produce excellent results. (In a later chapter, we'll see how having members interact changes things, sometimes for the better, sometimes for the worse.) Second, the group's guess will not be better than that of every single person in the group each time. In many (perhaps most) cases, there will be a few people who do better than the group. This is, in some sense, a good thing, since especially in situations where there is an incentive for doing well (like, say, the stock market) it gives people reason to keep participating. But there is no evidence in these studies that certain people consistently outperform the group. In other words, if you run ten different jelly-bean-counting experiments, it's likely that each time one or two students will outperform the group. But they will not be the same students each time. Over the ten experiments, the group's performance will almost certainly be the best possible. The simplest way to get reliably good answers is just to ask the group each time.

A similarly blunt approach also seems to work when wrestling with other kinds of problems. The theoretical physicist Norman L. Johnson has demonstrated this using computer simulations of individual "agents" making their way through a maze. Johnson, who does his work at the Los Alamos National Laboratory, was interested in understanding how groups might be able to solve problems that individuals on their own found difficult. So he built a maze--one that could be navigated via many different paths, some shorter, and some longer--and sent a group of agents into the maze one by one. The first time through, they just wandered around, the way you would if you were looking for a particular cafe* in a city where you'd never been before. Whenever they came to a turning point--what Johnson called a "node"--they would randomly choose to go right or left. Therefore some people found their way, by chance, to the exit quickly, others more slowly. Then Johnson sent the agents back into the maze, but this time he allowed them to use the information they'd learned on their first trip, as if they'd dropped bread crumbs behind them the first time around. Johnson wanted to know how well his agents would use their new information. Predictably enough, they used it well, and were much smarter the second time through. The average agent took 34.3 steps to find the exit the first time, and just 12.8 steps to find it the second.

The key to the experiment, though, was this: Johnson took the results of all the trips through the maze and used them to calculate what he called the group's "collective solution." He figured out what a majority of the group did at each node of the maze, and then plotted a path through the maze based on the majority's decisions. (If more people turned left than right at a given node, that was the direction he assumed the group took. Tie votes were broken randomly.) The group's path was just nine steps long, which was not only shorter than the path of the average individual (12.8 steps), but as short as the path that even the smartest individual had been able to come up with. It was also as good an answer as you could find. There was no way to get through the maze in fewer than nine steps, so the group had discovered the optimal solution. The obvious question that follows, though, is: The judgment of crowds may be good in laboratory settings and classrooms, but what happens in the real world?


II


At 11:38 am on January 28, 1986, the space shuttle Challenger lifted off from its launch pad at Cape Canaveral. Seventy-four seconds later, it was ten miles high and rising. Then it blew up. The launch was televised, so news of the accident spread quickly. Eight minutes after the explosion, the first story hit the Dow Jones News Wire.

The stock market did not pause to mourn. Within minutes, investors started dumping the stocks of the four major contractors who had participated in the Challenger launch: Rockwell International, which built the shuttle and its main engines; Lockheed, which managed ground support; Martin Marietta, which manufactured the ship's external fuel tank; and Morton Thiokol, which built the solid-fuel booster rocket. Twenty-one minutes after the explosion, Lockheed's stock was down 5 percent, Martin Marietta's was down 3 percent, and Rockwell was down 6 percent.

Morton Thiokol's stock was hit hardest of all. As the finance professors Michael T. Maloney and J. Harold Mulherin report in their fascinating study of the market's reaction to the Challenger disaster, so many investors were trying to sell Thiokol stock and so few people were interested in buying it that a trading halt was called almost immediately. When the stock started trading again, almost an hour after the explosion, it was down 6 percent. By the end of the day, its decline had almost doubled, so that at market close, Thiokol's stock was down nearly 12 percent. By contrast, the stocks of the three other firms started to creep back up, and by the end of the day their value had fallen only around 3 percent.

What this means is that the stock market had, almost immediately, labeled Morton Thiokol as the company that was responsible for the Challenger disaster. The stock market is, at least in theory, a machine for calculating the present value of all the "free cash flow" a company will earn in the future. (Free cash flow is the money that's left over after a company has paid all its bills and its taxes, has accounted for depreciation, and has invested in the business. It's the money you'd get to take home and put in the bank if you were the sole owner of the company.) The steep decline in Thiokol's stock price--especially compared with the slight declines in the stock prices of its competitors--was an unmistakable sign that investors believed that Thiokol was responsible, and that the consequences for its bottom line would be severe.

As Maloney and Mulherin point out, though, on the day of the disaster there were no public comments singling out Thiokol as the guilty party. While the New York Times article on the disaster that appeared the next morning did mention two rumors that had been making the rounds, neither of the rumors implicated Thiokol, and the Times declared, "There are no clues to the cause of the accident."

Regardless, the market was right. Six months after the explosion, the Presidential Commission on the Challenger revealed that the O-ring seals on the booster rockets made by Thiokol--seals that were supposed to prevent hot exhaust gases from escaping--became less resilient in cold weather, creating gaps that allowed the gases to leak out. (The physicist Richard Feynman famously demonstrated this at a congressional hearing by dropping an O-ring in a glass of ice water. When he pulled it out, the drop in temperature had made it brittle.) In the case of the Challenger, the hot gases had escaped and burned into the main fuel tank, causing the cataclysmic explosion. Thiokol was held liable for the accident. The other companies were exonerated.

In other words, within a half hour of the shuttle blowing up, the stock market knew what company was responsible. To be sure, this was a single event, and it's possible that the market's singling out of Thiokol was just luck. Or perhaps the company's business seemed especially susceptible to a downturn in the space program. Possibly the trading halt had sent a signal to investors to be wary. These all are important cautions, but there is still something eerie about what the market did. That's especially true because in this case the stock market was working as a pure weighing machine, undistorted by the factors--media speculation, momentum trading, and Wall Street hype--that make it a peculiarly erratic mechanism for aggregating the collective wisdom of investors. That day, it was just buyers and sellers trying to figure out what happened and getting it right.

How did they get it right? That's the question that Maloney and Mulherin found so vexing. First, they looked at the records of insider trades to see if Thiokol executives, who might have known that their company was responsible, had dumped stock on January 28. They hadn't. Nor had executives at Thiokol's competitors, who might have heard about the O-rings and sold Thiokol's stock short. There was no evidence that anyone had dumped Thiokol stock while buying the stocks of the other three contractors (which would have been the logical trade for someone with inside information). Savvy insiders alone did not cause that first-day drop in Thiokol's price. It was all those investors--most of them relatively uninformed--who simply refused to buy the stock.

But why did they not want Thiokol's stock? Maloney and Mulherin were finally unable to come up with a convincing answer to that question. In the end, they assumed that insider information was responsible for the fall in Thiokol's price, but they could not explain how. Tellingly, they quoted the Cornell economist Maureen O'Hara, who has said, "While markets appear to work in practice, we are not sure how they work in theory."

Maybe. But it depends on what you mean by "theory." If you strip the story down to its basics, after all, what happened that January day was this: a large group of individuals (the actual and potential shareholders of Thiokol's stock, and the stocks of its competitors) was asked a question--"How much less are these four companies worth now that the Challenger has exploded?"--that had an objectively correct answer. Those are conditions under which a crowd's average estimate--which is, dollar weighted, what a stock price is--is likely to be accurate. Perhaps someone did, in fact, have inside knowledge of what had happened to the O-rings. But even if no one did, it's plausible that once you aggregated all the bits of information about the explosion that all the traders in the market had in their heads that day, it added up to something close to the truth. As was true of those who helped John Craven find the Scorpion, even if none of the traders was sure that Thiokol was responsible, collectively they were certain it was.

What People are Saying About This

Kenneth Arrow

It has become increasingly recognized that the average opinions of groups is frequently more accurate than most individuals in the group. As a special case, economists have spoken of the role of markets in assembling dispersed information. The author has written a most interesting survey of the many studies in this area and discussed the limits as well as the achievements of self-organization.
winner of the Nobel Prize in Economics and Professor of Economics (Emeritus), Stanford University

Po Bronson

This book should be in every thinking businessperson's library. Without exception. At a time when corporate leaders have shown they're not always deserving of our trust, James Surowiecki has brilliantly revealed that we can trust each other. That we count. That our collective effort is far more important than the lofty predictions of those CEO-kings we have worshipped for too long.
author of What Should I Do With My Life?

Malcolm Gladwell

The Wisdom of Crowds is dazzling. It is one of those books that will turn your world upside down. It's an adventure story, a manifesto, and the most brilliant book on business, society, and everyday life that I've read in years.
author of The Tipping Point

Interviews

Some years ago, a professor at the University of Southern California named Jack Treynor began to do a curious thing: he would bring a jar of jellybeans to class and ask his students to guess just how many jellybeans the jar contained. The guesses ranged widely -- some students guessed high, some guessed low, and a few got close to the actual number of beans in the jar. The remarkable thing, though, was this: whenever Treynor ran this experiment, the class's average guess was close to perfect, and was better than the individual guesses of nearly every student in the class. On a day when the jar held 850 beans, for instance, the class' guess was 871, off by just two percent. And out of 56 students, only one made a better guess.

This was no party trick. Instead, the jellybean-counting experiment demonstrated something I call the wisdom of crowds: under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them. A group does not need to be dominated by exceptionally intelligent people in order to be smart. Even if most of the people within a group are not especially well-informed or rational, collectively they can still be brilliant.

The wisdom of crowds is at work all around us, although it's easy to miss. It helps explain why the economy works well at getting people what they want as cheaply as possible, even though no one is in charge of the economy as a whole. It helps explain why people in a healthy society cooperate with each other by doing things like paying taxes and contributing to charities. And it explains why, if you're trying to solve a problem with a definitive answer, you're better off with the group than with even the most well-informed expert.

Take the show Who Wants to Be a Millionaire. When a contestant on the show is stumped by a question, he can ask the audience for help or consult someone he's designated as an expert. The experts do a reasonable job: they get the answer right 65 percent of the time. But the audience is close to perfect: it gets the answer right 91 percent of the time. Similarly, at the race track, the final odds on a race reliably predict the race's order of finish (that is, the favorite wins most often, the horse with the second-lowest odds wins second most often, and so on) and also uncannily predict how likely it is that a horse will win. In other words, horses that run at 3-to-1 odds win just about a quarter of the time. The Hollywood Stock Exchange -- a kind of online stock market where people can wager on how movies will do at the box office -- has become a more reliable forecaster of ticket sales than any other method. And in the stock market, for all of its crazy swings, it's very difficult for even the best money managers to do better than the market as a whole.

So what's the catch? Simply this: not all crowds are wise. Groups can only be smart if they're made up of people with diverse information who are making decisions relatively independently. Groups that are too much alike get stuck because they don't have access to enough information and because they favor harmony over dissent. The wisdom of crowds isn't about consensus or compromise, so groups in which everyone agrees are likely to be less intelligent. Similarly, when people are worried too much about what others think, groups get dumber, too. The real paradox of the wisdom of crowds is that the way for the group to be smart is for each person to act as an individual. When they don't -- like during a stock-market bubble, where people buy stocks only because they think they can dump them on someone else -- groups go wrong.

Wise crowds, then, are diverse, independent, and decentralized. As it happens, that means that one place that is tailor-made for the wisdom of crowds is the Internet. Because no one is in charge of the Net, and because it features a cacophony of diverse voices, with no one voice being loud enough to drown out others, the Internet's collective intelligence is potentially immense. One place you can see this is in the plethora of rating systems for everything from consumer electronics to -- as on this site -- books. Until recently, there was no way to aggregate the judgments of readers or TV watchers across the country, let alone the world. So most of the information a person got came from a relatively small circle of friends and acquaintances. Today, we can draw on a far more diverse and eclectic pool of information. To the degree that we do so, our decisions will be better as a result.

The most striking example of the wisdom of crowds on the Net is, of course, Google. Google is able to routinely survey 4.3 billion web pages and pick exactly the one that has the information that you are looking for. And it does this by relying on the collective intelligence of the Web. Roughly speaking, Google finds the right pages by asking Web page producers to vote on which other pages are most worthwhile, with each link to a page counting as a vote. Google is a republic, not a pure democracy; the more people that have linked to a given page, the more influence that page has on the group's final decision. But the principle is fundamentally democratic -- let the masses decide. Far more often than not, they decide wisely.

Most of us, whether as voters or investors or consumers or managers, believe that valuable knowledge is concentrated in a very few hands (or, rather, in a very few heads). We assume that the key to solving problems or making good decisions is finding that one right person who will have the answer. But Jack Treynor's jellybean experiment and Google's success should teach us a different lesson. When it comes to problem-solving, stop looking for that one right person. Ask the crowd instead. Chances are, it knows. James Surowiecki

Customer Reviews

Most Helpful Customer Reviews

See All Customer Reviews

Wisdom of Crowds 3.5 out of 5 based on 0 ratings. 34 reviews.
Guest More than 1 year ago
This well-written bestseller explores the apparent anomaly that crowds of nonexperts seem to be collectively smarter than individual experts or even small groups of experts. This basic insight is at the heart of contemporary financial investment theory, with its emphasis on the difficulty of outguessing the market. Beginning with British scientist Francis Galton's remarkable discovery in 1906 that a crowd of nonexperts proved surprisingly competent at guessing the weight of an ox, financial columnist and author James Surowiecki skillfully recounts experiments, discoveries and anecdotes that demonstrate productive group thinking. The concept does not come as news to anyone reasonably well read in modern financial literature, but we recommend this comprehensive, fresh presentation.
Guest More than 1 year ago
I had high expectations for this book because James Surowiecki's New Yorker column is usually so good. But THE WISDOM OF CROWDS is one of the most disappointing books I've read in years. (Indeed, I feel somewhat ripped off by having purchased it and devoted several hours to reading it.) The main problem with this book is that despite Surowiecki's often breathless tone, nothing he says is new. Every point he makes has been made many times before by many other writers. For instance, the key theme of his book is that groups can solve certain 'cognition problems' better than individuals. No kidding. Ever hear the phrase 'Two heads are better than one?' The thesis is so self-evident and widely-known that it comes with its own cliché! Yet Surowiecki devotes more than one-third of the book essentially to arguing that two people can solve a crossword puzzle faster than one person. Amazing, no? What's more, Surowiecki's central point about the power of 'collective intelligence' has long been a staple of business education. If you've ever taken an organizational behavior class, you've done the exercise where groups of varying sizes are stuck on a desert island with a dozen supplies -- and then each group must devise a solution for escaping the island using those supplies. Inevitably, the larger the group, the better the solution -- because larger groups reflect the accumulated experience and expertise of more people. (In other words, five heads are even better than two.) Want another example of how threadbare this idea is? Google the phrase 'none of us is as smart as all of us' - and you'll discover that Surowiecki's supposedly 'counterintuitive' notion has been talked about in business circles since Bill Gates was in short pants. If that weren't bad enough, the rest of the book -- particularly Suriowiecki's discussion of 'coordination,' his second 'stunning' insight--- is essentially a retread of arguments that have been made elsewhere for more than a decade. James Gleick made many of these points in CHAOS. Kevin Kelly said everything that Surowiecki says ten years ago in OUT OF CONTROL. Steven Johnson said it again four years ago in EMERGENCE. Howard Rheingold said lots of it last year in SMART MOBS. And Surowiecki's third argument -- that sometimes cooperation is preferable to competition -- is even older. Charles Darwin told us this in the 19th century! Indeed, there's an entire branch of evolutionary psychology devoted to studying cooperation. Just read Robert Wright's THE MORAL ANIMAL if you want a more thorough and engaging account of this point. If this book were an undergraduate term paper that summarized the self-evident and reviewed what others had already had said, I'd give it a B. But for book that costs 20 bucks from a writer who's obviously got some talent, I'd have to give THE WISDOM OF CROWDS an Incomplete. Please try again, James. But next time, try a lot harder.
Guest More than 1 year ago
This is one of the most entertaining and intellectually engaging books I've come across in a long while. Surowiecki has a gift for making complex ideas accessible, and he has a wonderful eye for the telling anecdote. His thesis about the intelligence of groups made up of diverse, independent decision-makers seems initially counterintuitive, but by the end of the book it seems almost obvious, because of all the evidence Surowiecki piles up on its behalf. The book does cover a lot of ground in not very much space, and the pace of the argument is at times too fast. But the throughline of the argument is almost always clear, and the stories Surowiecki tells are often memorable. The chapter on NASA's mismanagement of the Columbia mission and the tale of how a man named John Craven relied on collective wisdom to find a lost submarine are especially striking. This is one of those books that I expect people will still be talking about and referring to years or even decades from now. It's also a book that I hope will have a concrete impact on the way that people make decisions, since the implications of Surowiecki's argument are radical in the best way. All in all, a terrific read.
Guest More than 1 year ago
The crowd theory only works, and then only sometimes, in fields without rigor. Ten million people who have not studied calculus could not integrate a simple polynomial. How many would it take to come up with Maxwell's equations? The book is a poor answer to an unasked question.
Guest More than 1 year ago
This book is actually a fine example of the kind of 'cascading' systems talked about in the book. It is enormously overhyped and poorly argued. The author presents what amounts to a dumbed down explanation of game theory with pseudo-social science. Horrible book skip it. Excellent books that cover this topic (generall) and are accessible are Dixit and Nalebuff's _Thinking Strategically_ and Barabasi's _Linked_.
shenzi More than 1 year ago
This is an entertaining and engaging book. It's somewhat repetitive, yet Surowiecki uses a great variety of intriguing examples to prove his viewpoint.
Anonymous More than 1 year ago
M_L_Gooch_SPHR More than 1 year ago
I enjoyed The Wisdom of Crowds. I found the book to be well researched and well written. While the arguments for crowd wisdom can often be very complex, Surowiecki presents them in a clear fashion so that even numbskulls like me can get the message. And while I agree with his arguments and positions on the wisdom of crowds and found truth is each of the 12 chapters I do not believe that I am fully sold on this idea. Certainly not when we look at a variety of other arenas. The stupidity of crowds is why our nation as a republic. God forbid that we were a true democracy and forged our laws passed strictly on the crowd sentiment. I also didn’t find a lot of wisdom in the recent OWS movement. I’m not saying this from a political point of view as I do not care either way but this movement couldn’t muster enough cognitive power to formulate a simple goal, much less the strategy to achieve the objective. And finally, from a personal viewpoint, I have found through years of collective bargaining that the wisdom of crowds is usually absent from these groups of contract voters. I have seen more than one manufacturing facility closed by management because the crowd would not pass a labor contract over a minor detail. Nope, no wisdom here. I could go on but this is suffice to say that while the book is excellent, the reader would be advised to approach these theories with a wide open mind. I do recommend as a thought provoking well executed tome on a very important subject. Michael L. Gooch, SPHR – Author of Wingtips with Spurs
Camejo More than 1 year ago
Worth the read to understand how we have evolved as a society through the technology. Working in IT and developing tools for info sharing it gave me insights into how these same technologies have permeated into every day life. Like many books of it's genre it did stretch the points a bit and could have been more concise.
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Anonymous More than 1 year ago