Why was there a decrease in the number of ratings?

At various stages in the contest, Next Stop Design had some blatant instances
of people trying to game the contest. These folks would register several fake
email addresses and would give one design a bunch of 5s and all the designs
above it in the rankings a bunch of 1s. This certainly violated the terms of
use for the site (you can only register one account), and it violated the
spirit of the experiment.

When the contest closed, we went through a final time and found several more
people who had gamed the site. We removed these accounts and their votes when
we found them. We went through a pretty rigorous series of steps to determine
invalid accounts, and when in doubt, we erred on the side of not deleting
someone rather than deleting them. We tracked multiple IP addresses, then
audited votes from those IPs for gaming patterns (lots of 1s and 5s on
targeted designs). We then dove deeper into suspected accounts, noticing
obvious password patterns, registration times, email account naming patterns,
and instances where the city/country information gathered in the registration
process didn’t match the user’s geo-IP address in any way. We considered
patterns of comments on certain designs as well. In all of these cases, it was
fairly clear who the gamers were. The top 10 designs received especially close
scrutiny. In total, 20 different instances of gaming were found and removed,
accounting for some 4,200 votes that were cast. This accounted for more than
26% of the votes in the entire contest.

We recognize that this method is still not entirely airtight, and that people
will game an online contest if they are so inclined. But we think this is a
pretty thorough method to rule out gamers, and the winning designs especially
had a very diverse base of votes from a diverse set of hundreds of registered
users. We’re confident that the winners are deserving.

If the design you submitted now has fewer votes than it did before the contest
ended, it was because some gamers on the site had cast votes for your design.
These votes were removed as fraudulent.

Thanks to everyone for submitting and participating. We hope you enjoyed this
experiment. And if you can think of some better ways to make our system more
gamer-proof, we would love to hear your thoughts.


The Next Stop Design Team

Sponsored by the Utah Transit Authority, Federal Transit Authority and The University of Utah