Telescope Time without Tears - Numberphile
video description
Date: 2022-04-08
Related videos
Comments and reviews: 9
Insanity
I don't understand why people see this system as flawed. I think it's brilliant! It's a win-win system. If your paper gets pushed back then you at least got the chance to see the general opinion, meaning you have an idea on how to improve your research to make it more appealing to the scientific community next time, and it gives you an idea of priority. Yes your stuff is good, but maybe this guy research might be more beneficial, and might even help your research in the end minus the cost of hiring telescope time. I'm pretty sure it is not cheap. And it's not like if you fail it is the end of the world, you always get next time. I believe this system encourage improvement. I have work with Peer Review systems before in engineering field, and it is freaking awesome to see others work. It really help you to develop ideas.
reply
I don't understand why people see this system as flawed. I think it's brilliant! It's a win-win system. If your paper gets pushed back then you at least got the chance to see the general opinion, meaning you have an idea on how to improve your research to make it more appealing to the scientific community next time, and it gives you an idea of priority. Yes your stuff is good, but maybe this guy research might be more beneficial, and might even help your research in the end minus the cost of hiring telescope time. I'm pretty sure it is not cheap. And it's not like if you fail it is the end of the world, you always get next time. I believe this system encourage improvement. I have work with Peer Review systems before in engineering field, and it is freaking awesome to see others work. It really help you to develop ideas.
reply
Bouke
The incentives of all participants are not necessarily sufficiently aligned with the desired outcome. Consider the situation that everyone has submitted one paper and that the objective ranking is transitive. Further assume that everybody can assess whether a given paper is better or worse than one's own paper. Then it makes sense to trash papers that are better than your own, especially when you have a chance of winning. I think it would depend on the fraction of successful papers whether the incentives are sufficiently aligned to get an optimal result, with a lower fraction of successes resulting in better outcomes. As an economist, I wonder, why not have researchers put money where their mouths are? Let them pay and if a lot of money comes in, build new telescopes. If no money comes in, don't build new ones.
reply
The incentives of all participants are not necessarily sufficiently aligned with the desired outcome. Consider the situation that everyone has submitted one paper and that the objective ranking is transitive. Further assume that everybody can assess whether a given paper is better or worse than one's own paper. Then it makes sense to trash papers that are better than your own, especially when you have a chance of winning. I think it would depend on the fraction of successful papers whether the incentives are sufficiently aligned to get an optimal result, with a lower fraction of successes resulting in better outcomes. As an economist, I wonder, why not have researchers put money where their mouths are? Let them pay and if a lot of money comes in, build new telescopes. If no money comes in, don't build new ones.
reply
Peter
The problem is your system doesn't actually figure out what everyone thinks is the best proposal. It figures out what everyone things everyone else will think is the best proposal (because of the bit about penalizing those who rank things differently. This has the negative feature of preventing change and locking in assumptions, e. g, if it becomes generally known that everyone thinks type X proposals are rubbish then, even when everyone changes their mind and decides X proposals are great, you risk those proposals still getting pushed to the bottom of the list since everyone rates them based on their assumptions about how others will rate them.
Yes, on some sufficiently strong assumption of perfect knowledge by all participants that won't happen but I worry that it will when real humans vote.
reply
The problem is your system doesn't actually figure out what everyone thinks is the best proposal. It figures out what everyone things everyone else will think is the best proposal (because of the bit about penalizing those who rank things differently. This has the negative feature of preventing change and locking in assumptions, e. g, if it becomes generally known that everyone thinks type X proposals are rubbish then, even when everyone changes their mind and decides X proposals are great, you risk those proposals still getting pushed to the bottom of the list since everyone rates them based on their assumptions about how others will rate them.
Yes, on some sufficiently strong assumption of perfect knowledge by all participants that won't happen but I worry that it will when real humans vote.
reply
Maxim
This way may cause monopolizing of the research field by the majority. Let's imagine there are two research fields, the members of each consider their own field the best and do not much appreciate or understand the other field, although the scientific value of both fields is roughly the same. If by chance it happens that a 70% competitors are from the 1st field and the rest 30% from the 2nd, then the 2nd field will be suppressed. It is much more probable that their applications are reviewed from someone from the 1st field, and downgraded. Those from the 2nd field who reviewed applications from the 2nd field will be also penalized as their opinion is different from the 70% majority.
reply
This way may cause monopolizing of the research field by the majority. Let's imagine there are two research fields, the members of each consider their own field the best and do not much appreciate or understand the other field, although the scientific value of both fields is roughly the same. If by chance it happens that a 70% competitors are from the 1st field and the rest 30% from the 2nd, then the 2nd field will be suppressed. It is much more probable that their applications are reviewed from someone from the 1st field, and downgraded. Those from the 2nd field who reviewed applications from the 2nd field will be also penalized as their opinion is different from the 70% majority.
reply
wreynolds1995
I watched this video and a question immediately occurred to me. I then watched the extras video, and I still have the same question.
The situation itself seems very comparable to student applications to universities (replace -astronomers- with -students-, -committee- with -university faculty members- and -telescope time- with -degree places- and you get basically the same situation as far as I can see, and yet I can't imagine anyone ever deciding that the students should be in charge of who goes to which university. So my question is, what's the difference? Why is this a good idea in the astronomical community and yet a bad idea for students?
reply
I watched this video and a question immediately occurred to me. I then watched the extras video, and I still have the same question.
The situation itself seems very comparable to student applications to universities (replace -astronomers- with -students-, -committee- with -university faculty members- and -telescope time- with -degree places- and you get basically the same situation as far as I can see, and yet I can't imagine anyone ever deciding that the students should be in charge of who goes to which university. So my question is, what's the difference? Why is this a good idea in the astronomical community and yet a bad idea for students?
reply
Take
This caught my attention because it's the same schema we use in a monthly writing contest I participate in. If we get more than X entries, we'll spend a week ranking slates of entries in the method described in this video, then vote on the top third or so. Last month, we turned 42 entries into 15, for instance, and that's all that people had to read to vote on (because even 30 stories is a lot to read in a week) And we remove the need for penalizing by making the whole thing anonymous. Of course, the downside is we can call for new slates to rank again and again until we've read every entry, and the final round is just spend thumb-twiddling, but it works.
reply
This caught my attention because it's the same schema we use in a monthly writing contest I participate in. If we get more than X entries, we'll spend a week ranking slates of entries in the method described in this video, then vote on the top third or so. Last month, we turned 42 entries into 15, for instance, and that's all that people had to read to vote on (because even 30 stories is a lot to read in a week) And we remove the need for penalizing by making the whole thing anonymous. Of course, the downside is we can call for new slates to rank again and again until we've read every entry, and the final round is just spend thumb-twiddling, but it works.
reply
KC
When you first proposed the problem, I immediately assumed that your solution was the way it worked. I was surprised to find there is a peer review committee, but not so shocked at their ineffectiveness. The most surprising part is that this is a bunch of doctors and not one of them could see that their system was less effective than the obvious solution.
I will say you'd have to make sure exactly 7 people receive each document. If it were random, one lucky guy could get a score of 50 when graded by 45 people, someone else could get a 28 when graded by 5 people, and guess who gets the telescope.
reply
When you first proposed the problem, I immediately assumed that your solution was the way it worked. I was surprised to find there is a peer review committee, but not so shocked at their ineffectiveness. The most surprising part is that this is a bunch of doctors and not one of them could see that their system was less effective than the obvious solution.
I will say you'd have to make sure exactly 7 people receive each document. If it were random, one lucky guy could get a score of 50 when graded by 45 people, someone else could get a 28 when graded by 5 people, and guess who gets the telescope.
reply
Vegard
The penalty system only works as long as the big majority are honest and does their best. If a majority tries to play the system by downvoting competitors, the system collapses because all the dishonest people will agree that the biggest competitors (that is, the best applications) deserve the lowest scores
I also don't like the fact that it turns the competition into a game of -guess the order of the other applicants-. There are ways to penalize the ones who consistently disagree with others without opening for possible abuse, such as handing out warnings, making them give reasons and so on
reply
The penalty system only works as long as the big majority are honest and does their best. If a majority tries to play the system by downvoting competitors, the system collapses because all the dishonest people will agree that the biggest competitors (that is, the best applications) deserve the lowest scores
I also don't like the fact that it turns the competition into a game of -guess the order of the other applicants-. There are ways to penalize the ones who consistently disagree with others without opening for possible abuse, such as handing out warnings, making them give reasons and so on
reply
sgsawant
I find a large number comments bashing the system and it's making me a bit miserable. Is the concept of --opportunity cost-- lost on everyone? I suppose it would be nearly impossible to come up with a strategy which is game proof. The idea is that this is the best system we have. Unless you can propose a better way out I don't think you have much of a point here.
I have suffered similar face-palm moments when people bash democracy. The flaws of democracy are common knowledge by now. It's just that the alternatives of democracy are way worse (at least when seen over a long term.
reply
I find a large number comments bashing the system and it's making me a bit miserable. Is the concept of --opportunity cost-- lost on everyone? I suppose it would be nearly impossible to come up with a strategy which is game proof. The idea is that this is the best system we have. Unless you can propose a better way out I don't think you have much of a point here.
I have suffered similar face-palm moments when people bash democracy. The flaws of democracy are common knowledge by now. It's just that the alternatives of democracy are way worse (at least when seen over a long term.
reply
Add a review, comment
Other channel videos
