Over the past few weeks we have been thinking of implementing some kind of a rating system on Codechef. We would like your opinions on a few aspects of the rating system. Some of the areas on which we would like your help are :
Fixed points for problems vs. Variable points for problems.
As of now, all our problems have a value of 1 (other than the challenge problem where only the best submission gets a value of 1 and all others get a value relative to that). Should we have the same points for all problems (other than the challenge type) or should the points for each problem vary according to the number of submissions for that problem. Along with the contest problems, this could extend to practice problems as well. So, the points awarded for solving a problem X having N users who have submitted correct solutions could be 1/N or some variant of this. If a new correct solution for one of the problems were to be submitted, we would change the points awarded for that problem accordingly. So, if someone were to solve a lot of easy problems, he/she would get less points compared to someone who would solve a really tough problem having fewer correct submissions. Any suggestions on what parameters we could consider while determining the points awarded to a problem would be welcome.
Cummulative rating vs. Dynamic rating based on ranks in a contest.
Every user would have a rating which would depend on his / her performance in a contest. There are two methods that we feel could be used for this. The first would be that the rating of the user is simply the sum of the points of the problems he / she has solved in the contest. The other method would be to base his rating on his actual rank in the contest and his expected rank based on his performance in the previous contests. If a participant performs poorly compared to his expected ranking, he would experience a rating drop and if he performs better than his expected ranking, he would get an increase in rating. Again, if you have any suggestions on what parameters we can consider while calculating the ratings for a particular user, they would be helpful.
Looking forward to your replies