Rating and Ranking system on Codechef

Posted by Aniruddha on August 5th, 2009 Filed in Features View Comments

Greetings,

Over the past few weeks we have been thinking of implementing some kind of a rating system on Codechef. We would like your opinions on a few aspects of the rating system. Some of the areas on which we would like your help are :

Fixed points for problems vs. Variable points for problems.
As of now, all our problems have a value of 1 (other than the challenge problem where only the best submission gets a value of 1 and all others get a value relative to that). Should we have the same points for all problems (other than the challenge type) or should the points for each problem vary according to the number of submissions for that problem. Along with the contest problems, this could extend to practice problems as well. So, the points awarded for solving a problem X having N users who have submitted correct solutions could be 1/N or some variant of this. If a new correct solution for one of the problems were to be submitted, we would change the points awarded for that problem accordingly. So, if someone were to solve a lot of easy problems, he/she would get less points compared to someone who would solve a really tough problem having fewer correct submissions. Any suggestions on what parameters we could consider while determining the points awarded to a problem would be welcome.

Cummulative rating vs. Dynamic rating based on ranks in a contest.
Every user would have a rating which would depend on his / her performance in a contest. There are two methods that we feel could be used for this. The first would be that the rating of the user is simply the sum of the points of the problems he / she has solved in the contest. The other method would be to base his rating on his actual rank in the contest and his expected rank based on his performance in the previous contests. If a participant performs poorly compared to his expected ranking, he would experience a rating drop and if he performs better than his expected ranking, he would get an increase in rating. Again, if you have any suggestions on what parameters we can consider while calculating the ratings for a particular user, they would be helpful.

Looking forward to your replies :)

  • Share/Bookmark
  • http://www.codechef.com The Chef

    In regards to the variable point system we could also include such factors as: number of submissions it took to solve the problem, time it took to solve the problem, etc…

  • http://www.codechef.com The Chef

    In regards to the variable point system we could also include such factors as: number of submissions it took to solve the problem, time it took to solve the problem, etc…

  • Tanaeem Moosa

    I think prefer using something like 1/ln(N) rather than than going 1/N. 1/N falls too fast. Also something like 1/ln(N*M) can be used, where M is number of submission.

  • Tanaeem Moosa

    I think prefer using something like 1/ln(N) rather than than going 1/N. 1/N falls too fast. Also something like 1/ln(N*M) can be used, where M is number of submission.

  • http://www.cre8d-design.com Stephen Merriman

    I don’t think you should take number of submissions into account. All this will do is encourage people to create multiple accounts and thus submit everything correct the first time on one account. Better to encourage people to keep trying by the fact that solving the problem in the end is all that matters.

  • http://www.cre8d-design.com Stephen Merriman

    I don’t think you should take number of submissions into account. All this will do is encourage people to create multiple accounts and thus submit everything correct the first time on one account. Better to encourage people to keep trying by the fact that solving the problem in the end is all that matters.

  • Imran

    I think Dynamic rating is better option here.

    Suppose one user solve 3 problem continuously for about 50 contest. Then he will have 150 points himself.

    Now, after that 50 contest, there’s a new user who is capable of solving 5 problems in every contest,then it will take around 75 competition for the new user to match the first one (supposing both continues solving 3 and 5 problems) even though second user is better coder then first one.

    Dynamic rating would reflect a real time rating.

  • Imran

    I think Dynamic rating is better option here.

    Suppose one user solve 3 problem continuously for about 50 contest. Then he will have 150 points himself.

    Now, after that 50 contest, there’s a new user who is capable of solving 5 problems in every contest,then it will take around 75 competition for the new user to match the first one (supposing both continues solving 3 and 5 problems) even though second user is better coder then first one.

    Dynamic rating would reflect a real time rating.

  • Imran

    I think Dynamic rating is better option here.

    Suppose one user solve 3 problem continuously for about 50 contest. Then he will have 150 points himself.

    Now, after that 50 contest, there’s a new user who is capable of solving 5 problems in every contest,then it will take around 75 competition for the new user to match the first one (supposing both continues solving 3 and 5 problems) even though second user is better coder then first one.

    Dynamic rating would reflect a real time rating.

  • Imran

    I think Dynamic rating is better option here.

    Suppose one user solve 3 problem continuously for about 50 contest. Then he will have 150 points himself.

    Now, after that 50 contest, there’s a new user who is capable of solving 5 problems in every contest,then it will take around 75 competition for the new user to match the first one (supposing both continues solving 3 and 5 problems) even though second user is better coder then first one.

    Dynamic rating would reflect a real time rating.

  • http://cheatchef.com xxyyzz

    Isn’t this like very similar to Topcoder? :O
    I think SPOJ system is very good (which is what Codechef is based on)
    BTW, DirectiPlex sounds very similar to Googleplex… :P

  • http://cheatchef.com xxyyzz

    Isn’t this like very similar to Topcoder? :O
    I think SPOJ system is very good (which is what Codechef is based on)
    BTW, DirectiPlex sounds very similar to Googleplex… :P

  • Tomasz Czajka

    @Imran: for a total contest score, instead of taking the sum of scores it could be considered to do something like this: sum_{i=0}^N R^i * score_i / maxscore_i,

    where score_0 is the score in the latest contest, score_N is the score in the oldest contest, score_i = 0 if the contestant didn’t compete. R = 0.99.

    This way old contest scores lose value over time. If you consistently get maximum possible score, this will converge to 1/(1-R) = 100.

  • Tomasz Czajka

    @Imran: for a total contest score, instead of taking the sum of scores it could be considered to do something like this: sum_{i=0}^N R^i * score_i / maxscore_i,

    where score_0 is the score in the latest contest, score_N is the score in the oldest contest, score_i = 0 if the contestant didn’t compete. R = 0.99.

    This way old contest scores lose value over time. If you consistently get maximum possible score, this will converge to 1/(1-R) = 100.

  • Tomasz Czajka

    @Imran: for a total contest score, instead of taking the sum of scores it could be considered to do something like this: sum_{i=0}^N R^i * score_i / maxscore_i,

    where score_0 is the score in the latest contest, score_N is the score in the oldest contest, score_i = 0 if the contestant didn’t compete. R = 0.99.

    This way old contest scores lose value over time. If you consistently get maximum possible score, this will converge to 1/(1-R) = 100.

  • Srihari

    I would prefer having the ratings to be dependent on “speed” or “time taken” rather than the number of submissions. This would surely encourage atleast me to write faster algorithms and try to micro-optimize as much as possible and at the same time keep the interest going for an already solved problem. Ofcourse the major problem might be the variety of languages being used. But maybe it’ll also encourage people to learn/know/understand the best language for a particular problem too :) .
    Also I hope you will consider rating for “Challenge Problems” in a different way (as that of the score itself or rank).

    Looking forward for the rating system!

  • Srihari

    I would prefer having the ratings to be dependent on “speed” or “time taken” rather than the number of submissions. This would surely encourage atleast me to write faster algorithms and try to micro-optimize as much as possible and at the same time keep the interest going for an already solved problem. Ofcourse the major problem might be the variety of languages being used. But maybe it’ll also encourage people to learn/know/understand the best language for a particular problem too :) .
    Also I hope you will consider rating for “Challenge Problems” in a different way (as that of the score itself or rank).

    Looking forward for the rating system!

  • Srihari

    I would prefer having the ratings to be dependent on “speed” or “time taken” rather than the number of submissions. This would surely encourage atleast me to write faster algorithms and try to micro-optimize as much as possible and at the same time keep the interest going for an already solved problem. Ofcourse the major problem might be the variety of languages being used. But maybe it’ll also encourage people to learn/know/understand the best language for a particular problem too :) .
    Also I hope you will consider rating for “Challenge Problems” in a different way (as that of the score itself or rank).

    Looking forward for the rating system!

  • Srihari

    I would prefer having the ratings to be dependent on “speed” or “time taken” rather than the number of submissions. This would surely encourage atleast me to write faster algorithms and try to micro-optimize as much as possible and at the same time keep the interest going for an already solved problem. Ofcourse the major problem might be the variety of languages being used. But maybe it’ll also encourage people to learn/know/understand the best language for a particular problem too :) .
    Also I hope you will consider rating for “Challenge Problems” in a different way (as that of the score itself or rank).

    Looking forward for the rating system!

  • Keshav Dhandhania

    Just don’t include Competition performance, and practice room performance to the same ratings. Sounds unfair.

  • Keshav Dhandhania

    Just don’t include Competition performance, and practice room performance to the same ratings. Sounds unfair.

  • Keshav Dhandhania

    Just don’t include Competition performance, and practice room performance to the same ratings. Sounds unfair.

  • Keshav Dhandhania

    Just don’t include Competition performance, and practice room performance to the same ratings. Sounds unfair.

  • http://www.cre8d-design.com/ Stephen Merriman

    I don't think you should take number of submissions into account. All this will do is encourage people to create multiple accounts and thus submit everything correct the first time on one account. Better to encourage people to keep trying by the fact that solving the problem in the end is all that matters.

  • http://www.cre8d-design.com/ Stephen Merriman

    I don't think you should take number of submissions into account. All this will do is encourage people to create multiple accounts and thus submit everything correct the first time on one account. Better to encourage people to keep trying by the fact that solving the problem in the end is all that matters.

  • http://www.cre8d-design.com/ Stephen Merriman

    I don't think you should take number of submissions into account. All this will do is encourage people to create multiple accounts and thus submit everything correct the first time on one account. Better to encourage people to keep trying by the fact that solving the problem in the end is all that matters.

blog comments powered by Disqus

Recent Posts

Categories

Recent Comments

Recent Pictures

Blogroll

Archives

Company Blogs

Careers@Directi