I did some thinking... and I think I have an idea:
-First, assume there are four judges.
The entries should be split into four groups, and each group is randomly assigned a judge.
The judge need only judge all the levels in their group at first.
Then, the top 25% moves on, and is judged by the rest of the judges. From there, we decide the winners. They officially score whatever their average was.
The bottom 75%? they are not judged again. They are officially scored as their rank against others by the same judge (i.e. Noivern-32nd).
This system is very high precision at the very top- which is exactly what a contest needs. After that, it becomes very vague, which isn't too much of an issue.
The people who didn't do the best still get a general idea of how they did, which is nice.
Plus, how the groups are made has a lot of flexibility. We could group people:
-Randomly
-Against people with similar themes
-Away from people with similar themes
-As they would be best suited to the judge
-Upon request (but have the judges randomized) (say two users who aren't great designers want to be scored against eachother- there we go)
Plus the percent for moving on can change- 40% for a larger group, 10% for a smaller one (but it can't be too small, or some people who otherwise would've done great wouldn't make it past round 1).
What do you think?
Want to see my Super Mario Timeline?