Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
This is often the result of attempting to overoptimize a system. You can optimize a race car to a huge degree, because you know exactly what you want it to do.
You can't optimize a schooling system, because you don't know exactly what you want it to do. A little noise is a good thing, because the you want a little wiggle room for teachers to sidestep the dictums of education czars, and students to sidestep the dictums of teachers.
The Greeks solved this quite a few years ago, with sortition. Under sortition (injecting noise into elections - http://en.wikipedia.org/wiki/Sortition), Bush and Gore would have been forced to pay "paper, scissors, rock" for the presidency. Under the US's more pure democracy, they would have been tempted to make all kind of Faustian bargains with sordid players to nail down the last 0.01% of votes.
Randomization means that the last percent is just not worth chasing, so players in a competition won't be tempted to bend the rules for a tiny advantage.
The same process could be used for tests. If you allocate places in desirable courses (say medicine) randomly to anyone above a certain score, the top students won't bother drilling as hard just to get the top score.
Stocks are the same - quants wouldn't sweat timing as much if their placement in order books was randomized. It would be more efficient to pay attention to fundamental value than momentary fluctuations if they weren't guaranteed to make large profits on the momentary fluctuations. Some would still work on timing, but not as many.
Patents are just bad policy badly implemented at the moment, not over-optimized.
Good comment. I smiled at the sheer simplicity of injecting randomness to mitigate the >95% in order to deter exploitation. Of course it would be an interesting experiment to conduct and compare results against.
Also regarding your racecar example; they've gotten so good at making them go fast they actually are required to have restrictor plates (http://en.wikipedia.org/wiki/Restrictor_plate) so this fits the example of exploiting a well-defined goal to the point it's no longer competitive/safe/fun.
I find that quite amusing and cool. (I don't follow the sport I just like watching science-type shows on tv)
Very interesting comment, hadn't heard of Sortition before. However, I see a few practical problems with this.
First, it makes it harder to fix any errors/cheating. You can ask for a reevaluation of exam papers or a recount of the ballot if you felt that something shady happened, but then what about the randomization phase. In the admissions example for instance, if after a re-eval, the composition of students in the >95 percentile changed, do you hold a new randomization to make it fair for the new kids who crossed the threshold? If you do that, is that fair to those who previously got selected in the previous admissions ballot, and weren't second time lucky? Would this lead to all the "losers" in the lottery to ask for re-evals just so that the lottery phase happens again? (given a large number of students, if there is any subjectivity in the evaluation, every re-eval is likely to change the composition, even if by a small amount)
Secondly, the losers in the lottery are going to feel cheated and angry, even if all evaluation was objective, and there was [provably] no cheating. I think most humans prefer an objective, unambiguous way of competing with each other, even when we think that the evaluation metric is flawed. So the unlucky competitors will feel angry post-lottery, even if everything was done in a fair manner.
Finally, it increases the chances of cheating/gaming the system at the randomization phase i.e. the administrator can game the lottery. And if/when the administrators do cheat, it is also easier for them to hide their cheating since the anomaly can be hidden within the random luck factor.
The reevaluation issue could be solved if all students in the >95 percentile is assigned a random number (e.g between 0.0 and 1.0) and then the students with the highest numbers are selected.
If a re-eval happens, all students that were already assigned a random number since before will keep it, and new random numbers will only be given to the students that previously were below the threshold.
This will have the effect that the list of selected students will not change much even after a re-evaluation. Only the few with lowest numbers risk losing their place.
I don't think the core of the idea is literal randomness injection. Compare for a moment high precision measuring devices. Any decent measuring device comes with a manual describing the tolerances of the device, the error rate, the expected skew caused by error and so on. They may be highly accurate, and very precise, but they don't claim 100% accuracy. Yet, ideally, this should be a simple task -- the math says if we do X to make the device, there is only one solution and the device is perfect. We take pains with these devices to construct them perfectly, but errors still slip in - and we acknowledge it. To counter these errors we do statistical analysis on the machines' output, we scrutinize anomalous readings and discard or normalize data within the error range and otherwise account for errors.
Look now at the Bush/Gore election cited above, in light of this. A vote is simply a measurement of the people's desire for leader X or leader Y. This measurement is taken via a tool called the ballot. The ballot is for some reason taken as an ideal measuring device - for some reason we think it is a perfect tool of measurement. Yet, a huge chunk of the debate and contention of the scenario came out of problems with the ballot itself -- hanging chad, butterfly ballot confusion, and so on, are all measurement issues. At some point, we should have said "look this is just too close to call with our ballot measuring system, other measures or means should be used to determine the outcome".
The same is true of tests, they are an imperfect measuring system we are trying to treat as idea; so are many other systems you allude to.
The core idea here, is to stop treating our measurements as some sort of ideal measurement and instead look at alternate measures and methods once you get within the error range of the original measuring device -- at this point you are in the realm of "luck" anyway, because there are other unaccounted factors affecting the measurement anyway. Further, it has no more effect on cheating than treating an imperfect measure as ideal -- cheaters in both cases attempt to exploit unnoticable issues in the measurement. Statistical cheating detection already considers this sort of thing anyway, if the statsticians are any good.
As for the social situation you present -- this is a trickier problem, but I suspect a large portion of it could go away once proper understanding is given to people. A lot of the "unfair" complaints are essentially based around "the measure is perfect, so if I can tweak the value measured of my by a small bit, it has some fundamental implication". Essentially we need to train people on the difference between platonic ideals and the realities of the scenario. Then instead of complaining over obscure minutia, energy can be focused on either bettering the overall situation, or bettering the measurement.
As to your final point: ignoring the imperfectness and pretending it is perfect does not really change the gaming problem, in fact it adds to it, because when you treat the imperfect as perfect, you must have lots of complicated rules, which also allow for crazy gaming. Instead effort could be better spent on gaming detection or simple administrator investigation/oversight.
The writer of the article is asking generally about how a system's rules can be exploited to the point that the original intention of the system is no longer achievable. The sortition is also a system with rules, therefore it can be exploited. As soon as I saw the rock-paper-scissors, I remembered a competition some time ago where different bots played rock-paper-scissors against each other. Now the
best bot exploited the reality of the seed number generators, compiled a database of sequences to find which seed the opponent was using, and so on.
But the idea is that it's not easy to exploit. If it is more than trivially difficult to leverage, then that is wasted time/effort that dilutes from the greater message.
ie, why waste effort on the "tie-breaker" message when you can just spend that effort avoiding it altogether and push yourself beyond the margin-of-error?
The key to sortition is that effort expended on gaming sortition is wasted and non-complementary effort compared to the original contest/competition... right now, getting that 0.1% by dirty tricks is equivalent to getting-out-the-vote of 0.1% more of the populace.
I don't think it is a good idea at all. If the political power boiled down to picking paper over scissors, the loosing side may very well ascribed no legitimacy to the outcome. That is almost certainly going to end far worse than actually counting all the votes (i am looking at you, supreme court).
The same problem happens with your exam example. Why is it fair that someone less competent than you got in? A better solution would be to let everyone over the cut off point in.
I can't disagree with your example of the stock market, so long as the trades were still executed with a few seconds.
If some guy gets into medicine because their parents did their homework, drilled them in SAT techniques, and hired a firm to fabricate a good admissions letter, are they really more competent than a kid who did their own homework, got a high SAT score because of their IQ and good study habits, and wrote an honest admissions letter?
Do people think our current politicians are legitimate? I'd rather see them be more willing to say what they think, and act with conviction than get a guy who is 0.1% more popular.
Yeah, it's a strawman. Will randomization discourage people from playing the game dishonestly? Or will people panic, and play the game harder, and give up on the honest component of what they are doing? I don't really know.
It would just be recognizing the jitter that exists inherently in the system. You will always get a different number when counting the votes, so you can end up with a "50% voted for both" situation.
Why is flipping a coin worse than pretending the next count will be more correct? You kind of contradict your own point by implying the supreme court didn't supply legitimacy to the election. Nobody can blame a quarter for being stiffed with neocons on the heads side.
The trouble is that getting people to recognize that is really hard, and just changing how the system works probably won't do it. If you want to politically antagonize nearly any American with a strong political affiliation, regardless of what that affiliation is, just tell them that the margin of error in the 2000 presidential elections was far larger than the difference in votes, and therefore both candidates were about equally legitimate winners.
The losing side will always find some reason to claim the outcome is illegitimate and if you don't get in you will always be able to find someone you think you are better than.
What might make sense would be a combination. Have some barrier to entrance to limit the people to choose from, but then select randomly from the remaining candidates. For example, to be considered for an elected position, you first have to receive %x of the vote, or maybe only select from the top 3. This way you help to ensure that the person being elected is representing some of the population, but hopefully eliminating some of the games.
Interesting. Somewhat akin to dithering, i.e. deliberately inserting noise in quantized signals to increase their perceived quality (http://en.wikipedia.org/wiki/Dither).
Do you have any links on this? I hadn't heard of it before and am quite interested in it now, but could not come up with anything after a quick Google search for "sortation". Thanks for sharing this!
The main disadvantage with sortation seems to be that it killed Socrates. Randomly selected jurors did quite a few nasty and arbitrary things. But an elected tyrant (especially one who was willing to do anything to stay popular) might be worse. It's hard to say that sortation is bad because it allowed mistakes - every political system allows mistakes.
Also, systematic slavery, misogyny, the occasional genocide and other discrimination against non-male-citizens (non-voters) was a problem, but that's not really related to sortation. Any democracy can have those problems. Fortunately, we seem to have progressed a little in the last 2000 years (despite the occasional step back).
> The main disadvantage with sortation seems to be that it killed Socrates.
I was just reading this morning on the way to work a note made by Voltaire to Montesquieu's "De l'esprit des lois" that the decision in Socrate's trial was in fact a pretty close one, so things were not that bad (he still died, I know). I don't want to turn this into /r/politics, but look what's happening to Bradley Manning. I'm pretty sure that even if he were to be put to trial in front of a civilian court the call won't be as close as in the case of Socrates, for a lot of different reasons which I'm not going into details now.
> In the case of Socrates, the jury found Socrates guilty on a relatively close vote of 280 to 220. (Interestingly, if less than 100 jurors voted for guilt, the accusers had to pay a fine to cover trial costs.)
It's been argued in the past that Socrates quite deliberately provoked the jury into killing him - it's hard not to read it, see that far from being humble or defending himself he instead told them they should vote him a pension for his services to Athens, and think that this was just a case of suicide by cop.
I don't want to turn this into /r/politics, but look what's happening to Bradley Manning. I'm pretty sure that even if he were to be put to trial in front of a civilian court the call won't be as close as in the case of Socrates, for a lot of different reasons which I'm not going into details now.
Despite being a cause celebre, the fact remains that Manning's alleged actions were, in fact, illegal. Not that jury nullification doesn't happen, but it's actively discouraged.
> Randomly selected jurors did quite a few nasty and arbitrary things. But an elected tyrant (especially one who was willing to do anything to stay popular) might be worse. It's hard to say that sortation is bad because it allowed mistakes - every political system allows mistakes.
Exactly. The point of voting is not, repeat not to execute the will of "the people" as faithfully as possible. "The people" can be just as powerful and arbitrary as a single ruler, when it comes to that. The point of voting is to randomize so that no single entity stays in power indefinitely. (No executive, legislature, or even set of ideals can win elections indefinitely, no matter how popular they are when they start out.) In exchange for mediocre performance in the average case, you get to eliminate the almost unboundedly terrible performance of the worst case.
Apparently, the expression is 'sortition' not 'sortation'. wisty has a link to the relevant Wikipedia article: http://en.wikipedia.org/wiki/Sortition, and a Google search for 'sortition' yields quite a few results.
>>Randomization means that the last percent is just not worth chasing...
True, but only in the case of complete randomization. In the case of randomization kicking in only when the election is close, one side will be chasing the last percent to get to randomization and the other to avoid it. Same thing...
Maybe the randomization could be weighted by the individual percentages. For example, randomize if the candidates are within 1%, but give the higher-voted candidate an increased chance of winning weighted by how far apart they are. That way the difference kicks in slowly, reducing the incremental value of additional votes in a reasonably smooth fashion.
Over-optimization of race cars can be bad too. Take the recent popularity of vintage racing. People don't want to watch 50 similar-looking, generic, computer-designed, air-tunnel-tested, same-engine cars going around a round track.
What they want to see is unique and historical body styles for every company in the race, powerful cars vs. nimble cars, finicky engines, twisty turns and long straightaways, etc.
Could be an example in the original post, really. Over-optimization leads to a worse sport.
We had something on here many months ago about a voting system basically the same as what we have now, however instead of determining the winner by who got the most votes, the winner is determined by randomly choosing a single ballot.
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
This is often the result of attempting to overoptimize a system. You can optimize a race car to a huge degree, because you know exactly what you want it to do.
You can't optimize a schooling system, because you don't know exactly what you want it to do. A little noise is a good thing, because the you want a little wiggle room for teachers to sidestep the dictums of education czars, and students to sidestep the dictums of teachers.
The Greeks solved this quite a few years ago, with sortition. Under sortition (injecting noise into elections - http://en.wikipedia.org/wiki/Sortition), Bush and Gore would have been forced to pay "paper, scissors, rock" for the presidency. Under the US's more pure democracy, they would have been tempted to make all kind of Faustian bargains with sordid players to nail down the last 0.01% of votes.
Randomization means that the last percent is just not worth chasing, so players in a competition won't be tempted to bend the rules for a tiny advantage.
The same process could be used for tests. If you allocate places in desirable courses (say medicine) randomly to anyone above a certain score, the top students won't bother drilling as hard just to get the top score.
Stocks are the same - quants wouldn't sweat timing as much if their placement in order books was randomized. It would be more efficient to pay attention to fundamental value than momentary fluctuations if they weren't guaranteed to make large profits on the momentary fluctuations. Some would still work on timing, but not as many.
Patents are just bad policy badly implemented at the moment, not over-optimized.