Sweet Potatoes,
The July contest is officially closed. While we are calculating the results, we wanted to share some feedback and ideas for improvement for the next contest:
- We need one or two harder questions. As many of you know, it is difficult to come up with challenging problems. If anyone is interested in setting problems for future contest, get in touch with us at [email protected] – we will pay for any problems we use.
- The contest needs to be shorter. Next month’s contest will be one week long. We will also be including a shorter duration contest at the end of next month (a few hours to a weekend – still deciding).
- We need to figure out a better way to determine winners for the challenge. Rejudging all submissions with new test data doesn’t make sense, you can submit a bunch of variations which will perform differently for different input data. There are two approaches we are considering: either limit the number of successful submissions one can submit for a problem (to let’s say 15) and not rejudge, only rejudge the last few accepted solutions (say 5), or only allow a certain number of submissions to the challenge problem a day (let’s say 10). Your thoughts on this appreciated.
- We need to define rules better. There can be no collaboration involved between contestants. Any form of collaboration (whether it’s reviewing code, asking for output for a set of inputs, or asking for time complexity of an accepted solution) is not allowed. We will immediately ban you on suspicion. More on this in a subsequent post.
If anyone has any other suggestions on improvements for the contest format, types of problems, new competitions, etc… please let us know.
Cheers,
Chefarino
Chef : The contest needs to be shorter. Next month’s contest will be one week long. We will also be including a shorter duration contest at the end of next month (a few hours to a weekend – still deciding).
Rahul : Well , honestly I wouldn’t want the contest to be shorter for I am a novice and I found the longer duration of the competition to be an advantage for me , not just in getting my submissions accepted but in improving my algorithmic skills as well . My general scenario was reading the problem statement and then making a brute force solution thus getting TLE , then trying to optimize it and still get another TLE . Then however I would search Google for new and better algorithms and try to implement them and keep on doing this until I get a solution accepted . During the June Algorithm Challenge , I had to submit nearly 50 submissions per problem to get my solution accepted and at least 10 of them had different strategies and various had different types of optimizations thus explaining and answering some of my questions as to which solutions are better and why . Most of my time during June Algorithm Challenge , in fact all of it , was gone into thinking about “Why did that Algorithm gave an TLE ” and thus learning about new algorithms . Even after the contest , most of my time was spent in learning new algorithms , strategies etc and solving problems at other judges like UVa and I am able to see some improvement in me . No more of my programs are getting TLEs at other judges now . So I would really really wish for a longer contest .
Chef : The contest needs to be shorter. Next month’s contest will be one week long. We will also be including a shorter duration contest at the end of next month (a few hours to a weekend – still deciding).
Rahul : Well , honestly I wouldn’t want the contest to be shorter for I am a novice and I found the longer duration of the competition to be an advantage for me , not just in getting my submissions accepted but in improving my algorithmic skills as well . My general scenario was reading the problem statement and then making a brute force solution thus getting TLE , then trying to optimize it and still get another TLE . Then however I would search Google for new and better algorithms and try to implement them and keep on doing this until I get a solution accepted . During the June Algorithm Challenge , I had to submit nearly 50 submissions per problem to get my solution accepted and at least 10 of them had different strategies and various had different types of optimizations thus explaining and answering some of my questions as to which solutions are better and why . Most of my time during June Algorithm Challenge , in fact all of it , was gone into thinking about “Why did that Algorithm gave an TLE ” and thus learning about new algorithms . Even after the contest , most of my time was spent in learning new algorithms , strategies etc and solving problems at other judges like UVa and I am able to see some improvement in me . No more of my programs are getting TLEs at other judges now . So I would really really wish for a longer contest .
Being a novice when it comes to algorithms I somehow prefer the longer formats.. 🙂 seems to give me more time to get my answers in and feel competitive.
How about having all 3 forms: (a) marathon aka (tests in cricket), (b) current formats (one week long) aka one dayers and (c) 20-20s (3+ hours long).
Personally wouldn’t like restriction on number of submissions, but taking only the last 5 entries or letting 5 entries to be nominated would be preferable though.
Being a novice when it comes to algorithms I somehow prefer the longer formats.. 🙂 seems to give me more time to get my answers in and feel competitive.
How about having all 3 forms: (a) marathon aka (tests in cricket), (b) current formats (one week long) aka one dayers and (c) 20-20s (3+ hours long).
Personally wouldn’t like restriction on number of submissions, but taking only the last 5 entries or letting 5 entries to be nominated would be preferable though.
I agree completely with Srihari and Rahul. Put in the tougher problems but do keep the contests long. Not everyone is as dedicated towards solving such problems as the experts. By making it shorter, you’ll lose the fun in the contest. A large part of the joy is discussing the problems with others and seeing their approaches. You wouldn’t like losing the novice community which feels encouraged by the longer timeframe. Seeing other (non-genious contestants) solve problems after persisting with their efforts gives motivation to try. Reducing the contest duration is bad in my opinion.
Consequently, restricting the number of submissions per day is counterintuitive. Rejudge the last few submissions but do keep the number of submissions unlimited.
I agree completely with Srihari and Rahul. Put in the tougher problems but do keep the contests long. Not everyone is as dedicated towards solving such problems as the experts. By making it shorter, you’ll lose the fun in the contest. A large part of the joy is discussing the problems with others and seeing their approaches. You wouldn’t like losing the novice community which feels encouraged by the longer timeframe. Seeing other (non-genious contestants) solve problems after persisting with their efforts gives motivation to try. Reducing the contest duration is bad in my opinion.
Consequently, restricting the number of submissions per day is counterintuitive. Rejudge the last few submissions but do keep the number of submissions unlimited.
I’d just like to say that I was a little disappointed from this month’s contest. If you are going to give problems which belong to a 5 hour contest in a 13-day contest, I guess there is no point. Why not just give a tie breaker and ask everyone to solve it, the other problems wont come into picture anyway.
The last month contest was so much better than this one.
Also, as I guess already pointed out by someone before, that rejudging causes heavy load on server and some acc solutions TLE. What if one of the TLE solution if submitted in normal conditions would have acc’d. This way you will clearly change the winners of the contest and the deserving person may not get their win. I guess you must either find a better way of rejudging or probably look towards making the test data better and not have a rejudge in the end. And anyway if your test data is good, people wont be able to take advantage of test data by pruning their solution just to the test data.
I think rejudging only a few solutions out of all is not a good idea, but maybe its just me. You can ask for opinion of others in this matter.
I’d just like to say that I was a little disappointed from this month’s contest. If you are going to give problems which belong to a 5 hour contest in a 13-day contest, I guess there is no point. Why not just give a tie breaker and ask everyone to solve it, the other problems wont come into picture anyway.
The last month contest was so much better than this one.
Also, as I guess already pointed out by someone before, that rejudging causes heavy load on server and some acc solutions TLE. What if one of the TLE solution if submitted in normal conditions would have acc’d. This way you will clearly change the winners of the contest and the deserving person may not get their win. I guess you must either find a better way of rejudging or probably look towards making the test data better and not have a rejudge in the end. And anyway if your test data is good, people wont be able to take advantage of test data by pruning their solution just to the test data.
I think rejudging only a few solutions out of all is not a good idea, but maybe its just me. You can ask for opinion of others in this matter.
Also, how about having a team contest ? :). For those of us who dont like the individual format that much
Also, how about having a team contest ? :). For those of us who dont like the individual format that much
I like the idea of limiting the number of submissions to 15 (or 100) instead of rejudging on new test data.
Make sure the inputs are large so that not much specific information can be gathered with the submissions. The July problem was good – large data, can’t “download” it or get much useful information via submissions (maybe should have more than 9 test cases).
One minor comment: it’s not clear what the memory limit is – issues like this would not be such a big problem though if rejudging goes away, because contestants would immediately know if they allocated too much memory or not.
I like the idea of limiting the number of submissions to 15 (or 100) instead of rejudging on new test data.
Make sure the inputs are large so that not much specific information can be gathered with the submissions. The July problem was good – large data, can’t “download” it or get much useful information via submissions (maybe should have more than 9 test cases).
One minor comment: it’s not clear what the memory limit is – issues like this would not be such a big problem though if rejudging goes away, because contestants would immediately know if they allocated too much memory or not.
Let me also add that I find it not a bad thing, but a *good* thing if contestants can try submitting something to gather some information about the inputs, maybe some statistics. It makes the contest a little interactive, the scoreboard is real, the task and results are clear.
Otherwise, if you rejudge, contestants may have to guess what kind of input the judges will provide, the final ranking may depend on the choice of unknown inputs.
Let me also add that I find it not a bad thing, but a *good* thing if contestants can try submitting something to gather some information about the inputs, maybe some statistics. It makes the contest a little interactive, the scoreboard is real, the task and results are clear.
Otherwise, if you rejudge, contestants may have to guess what kind of input the judges will provide, the final ranking may depend on the choice of unknown inputs.
“limit the number of successful submissions one can submit for a problem (to let’s say 15)”
I will point out that it’s also possible to gather information from unsuccessful submissions. Hence, I think a limit on total submissions (successful or not) would be better, but probably larger than 15.
“limit the number of successful submissions one can submit for a problem (to let’s say 15)”
I will point out that it’s also possible to gather information from unsuccessful submissions. Hence, I think a limit on total submissions (successful or not) would be better, but probably larger than 15.
Just one quick suggestion: Try to create “new” problems. Some of the problems in this contest were almost exact matches to other contest problems (ACM, USACO). This puts people who have seen the problems at an advantage, and if the problems are archived, people can ask “innocently” about that problem on the ACM site when their true motivation is the contest problem.
Just one quick suggestion: Try to create “new” problems. Some of the problems in this contest were almost exact matches to other contest problems (ACM, USACO). This puts people who have seen the problems at an advantage, and if the problems are archived, people can ask “innocently” about that problem on the ACM site when their true motivation is the contest problem.
Maybe I don’t understand the spirit of the challenge problem, but I think it would make the most sense to rejudge the last submission (because the programmer should be confident in his/her changes made to the program).
Maybe I don’t understand the spirit of the challenge problem, but I think it would make the most sense to rejudge the last submission (because the programmer should be confident in his/her changes made to the program).
I thought the length of the contest was fine. Although I solved all the problems within the first few days, I still had the challenge problem to work on.
I think rejudging the last 5 (or some fixed number) of submissions to the challenge problem is a good solution. Limiting the frequency of submissions would be another reasonable solution. I don’t like a total limit on the number of submissions (successful or otherwise) for the simple reason that it took me about 15 submissions this time just to figure out that the problem statement was incorrect. I also think that there should be more cases, at least for the rejudging.
I thought the length of the contest was fine. Although I solved all the problems within the first few days, I still had the challenge problem to work on.
I think rejudging the last 5 (or some fixed number) of submissions to the challenge problem is a good solution. Limiting the frequency of submissions would be another reasonable solution. I don’t like a total limit on the number of submissions (successful or otherwise) for the simple reason that it took me about 15 submissions this time just to figure out that the problem statement was incorrect. I also think that there should be more cases, at least for the rejudging.
Which ones are from ACM and which from USACO? Can you clarify before making accusations?
Which ones are from ACM and which from USACO? Can you clarify before making accusations?
1. I feel that the duration of the contest was quite appropriate. For example, in my case I only have weekends and a few hours afteroffice to have a look at the problems and try them out. In addition, I don’t think that these problems were “easy” or “obvious” to solve. May be I lack experiences from ACM or USACO, but except for two/three problems, all others have taken me more than 10 to 12 hours to crack. (eg: For somebody who doesnt know anything related to “quadratic reciprocity”, the quadratic equation problem is certainly not a “trivial” problem). So restricting duration of competition is a very bad idea.
2. I look at these competitions from a different angle – They are giving me a very good opportunity which TopCoder or any other short duration competitions don’t give – I have ample time to solve a problem which lets me learn a lot or things in the process. This is really a value add, and looking other people crack the problems after 5/10/20/50 wrong attempts is also a very good inspiration. For example, I would have never sat down and coded a convex-hull algo myself if it would not have been in this contest.
1. I feel that the duration of the contest was quite appropriate. For example, in my case I only have weekends and a few hours afteroffice to have a look at the problems and try them out. In addition, I don’t think that these problems were “easy” or “obvious” to solve. May be I lack experiences from ACM or USACO, but except for two/three problems, all others have taken me more than 10 to 12 hours to crack. (eg: For somebody who doesnt know anything related to “quadratic reciprocity”, the quadratic equation problem is certainly not a “trivial” problem). So restricting duration of competition is a very bad idea.
2. I look at these competitions from a different angle – They are giving me a very good opportunity which TopCoder or any other short duration competitions don’t give – I have ample time to solve a problem which lets me learn a lot or things in the process. This is really a value add, and looking other people crack the problems after 5/10/20/50 wrong attempts is also a very good inspiration. For example, I would have never sat down and coded a convex-hull algo myself if it would not have been in this contest.
3. There are many constructive ways to avoid the input-guessing problem that Codechef is currently facing, without limiting the contest time or number of submissions etc.
(a) Something similar to GCJ – Have two sets of data. The first set is just like the current set, whereas the second set is something against which the code can be tested exactly once (or may br 5 times at most) (per competitor). The coder can decide which solution should be tested against the second data set. Also, testing against second data set is not necessarily done at the end of the contest, but as soon as when a user submits his solution for the second data set. The solution would get the points only when it passes both sets.
(b) Keep 1 problem in the competition such that its result (whether the submission was correct or wrong) will *not* be disclosed during the contest. Thus, people would submit the solution but would never know if the answer was wrong or right till the competition ends. (This rule can be a bit relaxed so that if there are any issues- eg compilation error, memory error or TLE, then they will be shown. But cases like NZEC, wrong answer, correct answer, abort etc will not be disclosed)
(c) Keep 1 problem in the competition for which “test data” would be “open-for-contribution”. Any competitor would be allowed to submit his own test data for a problem, which in turn would be tested against all “AC” solutions, and all those which fail for that data will automatically disqualify. (of course the test data would be first exeucted on codechef’s own solution to generate the correct output). This is Something similar to “challenge phase” of TopCoder but without looking at other competitor’s code. One way to avoid participanst flooding codechef’s system by test data is to limit the number of test data one competitor can submit, and keep this phase open only for a short duration (say for final 2 days in a 10 day competition). Further, rules can be made a bit stringent by saying that a competitor is not allowed to resubmit a solution that fails on another competitor’s test data.
(d) Provide (very little) penalties for a wrong submission. For example, the score for an AC solution can be calculated as
1- 0.02*(number of attempts-10) so something similar. This automatically restricts the max number of attempts to 50,and also allows “first 10 attempts” without any penaulty.
3. There are many constructive ways to avoid the input-guessing problem that Codechef is currently facing, without limiting the contest time or number of submissions etc.
(a) Something similar to GCJ – Have two sets of data. The first set is just like the current set, whereas the second set is something against which the code can be tested exactly once (or may br 5 times at most) (per competitor). The coder can decide which solution should be tested against the second data set. Also, testing against second data set is not necessarily done at the end of the contest, but as soon as when a user submits his solution for the second data set. The solution would get the points only when it passes both sets.
(b) Keep 1 problem in the competition such that its result (whether the submission was correct or wrong) will *not* be disclosed during the contest. Thus, people would submit the solution but would never know if the answer was wrong or right till the competition ends. (This rule can be a bit relaxed so that if there are any issues- eg compilation error, memory error or TLE, then they will be shown. But cases like NZEC, wrong answer, correct answer, abort etc will not be disclosed)
(c) Keep 1 problem in the competition for which “test data” would be “open-for-contribution”. Any competitor would be allowed to submit his own test data for a problem, which in turn would be tested against all “AC” solutions, and all those which fail for that data will automatically disqualify. (of course the test data would be first exeucted on codechef’s own solution to generate the correct output). This is Something similar to “challenge phase” of TopCoder but without looking at other competitor’s code. One way to avoid participanst flooding codechef’s system by test data is to limit the number of test data one competitor can submit, and keep this phase open only for a short duration (say for final 2 days in a 10 day competition). Further, rules can be made a bit stringent by saying that a competitor is not allowed to resubmit a solution that fails on another competitor’s test data.
(d) Provide (very little) penalties for a wrong submission. For example, the score for an AC solution can be calculated as
1- 0.02*(number of attempts-10) so something similar. This automatically restricts the max number of attempts to 50,and also allows “first 10 attempts” without any penaulty.
Definitely if there is a limit, it should be much larger than 15 total submissions (otherwise people will run out even with reasonable fixes), I would make it more like 150. Limiting the frequency of submissions is almost the same thing – only your submissions have to be more spread out in time. Some kind of (big) limits might be useful regardless of other things to prevent “spamming” with submissions (even if not deliberately destructive).
Now I’m thinking that for the Lights problem 9 cases may have been enough, given that each case was huge – this way we got a nice 5 seconds / test case which probably wouldn’t be possible with more cases.
I don’t mind rejudging too much, but I don’t think it’s necessary for a good choice of problems, and immediate scoring has its own beauty. Maybe the scoring procedure should be decided on a case by case basis.
Definitely if there is a limit, it should be much larger than 15 total submissions (otherwise people will run out even with reasonable fixes), I would make it more like 150. Limiting the frequency of submissions is almost the same thing – only your submissions have to be more spread out in time. Some kind of (big) limits might be useful regardless of other things to prevent “spamming” with submissions (even if not deliberately destructive).
Now I’m thinking that for the Lights problem 9 cases may have been enough, given that each case was huge – this way we got a nice 5 seconds / test case which probably wouldn’t be possible with more cases.
I don’t mind rejudging too much, but I don’t think it’s necessary for a good choice of problems, and immediate scoring has its own beauty. Maybe the scoring procedure should be decided on a case by case basis.
Few important points that Codechef and problem setters should note-
(a) Make the format very clear, like TopCoder. Keep distinct sections like “Problem statement”, “Constraints”, “format of input”, “format of output” and “examples”. Make sure that you provide all necessary data and verify twice that there are no errors. It gives a very bad impression if it is found that the problem statement has errors.
(b) PLEASE be active on forums to answer the relevant queries and delete the possible hints / leakouts that other coders might knowingly/unknowingly post. For example, in my case, I asked for the upper limit on number of test cases for Johny the farmer problem but nobody even cared to reply. A simple reply like “This won’t be disclosed” was also acceptable, but not replying at all gives a feeling that admins are themselves ignorant.
Few important points that Codechef and problem setters should note-
(a) Make the format very clear, like TopCoder. Keep distinct sections like “Problem statement”, “Constraints”, “format of input”, “format of output” and “examples”. Make sure that you provide all necessary data and verify twice that there are no errors. It gives a very bad impression if it is found that the problem statement has errors.
(b) PLEASE be active on forums to answer the relevant queries and delete the possible hints / leakouts that other coders might knowingly/unknowingly post. For example, in my case, I asked for the upper limit on number of test cases for Johny the farmer problem but nobody even cared to reply. A simple reply like “This won’t be disclosed” was also acceptable, but not replying at all gives a feeling that admins are themselves ignorant.
@Tomek
Could you please suggest the novices , the order of online judges ,they should practice at ?
Eg UVa , SPOJ , USACO ,CodeChef etc
@Tomek
Could you please suggest the novices , the order of online judges ,they should practice at ?
Eg UVa , SPOJ , USACO ,CodeChef etc
It’d be great if the contest is shorter having duration of 5 hours.
Topcoder had a very successful TCO format on the same lines. Hope we can see same here, I had heard rumors that there would be something like Codechef Cup (similar to TCO)
That would be great. Further comments from Chef awaited 🙂
It’d be great if the contest is shorter having duration of 5 hours.
Topcoder had a very successful TCO format on the same lines. Hope we can see same here, I had heard rumors that there would be something like Codechef Cup (similar to TCO)
That would be great. Further comments from Chef awaited 🙂
1. I do not understand why we should limit the number of submissions when you are going to have rejudging too. When you have rejudging, you need not restrict submissions. For a longer format contest like this any sort of restriction will affect. You can choose to rejudge only the last submission or last 5 submissions. That is better.
2. We need tough problems. A couple of them at least. The June contest was perfect.
3. I am disappointed that the tie-breaker is deciding the winners this time. Let this not happen next time.
1. I do not understand why we should limit the number of submissions when you are going to have rejudging too. When you have rejudging, you need not restrict submissions. For a longer format contest like this any sort of restriction will affect. You can choose to rejudge only the last submission or last 5 submissions. That is better.
2. We need tough problems. A couple of them at least. The June contest was perfect.
3. I am disappointed that the tie-breaker is deciding the winners this time. Let this not happen next time.
Another Important point.
Please do not give such easy time constraints! Only the best algorithm should pass.
Let me give 2 examples,
1. Lights Off – Here n^6 algorithm was passing without optimization when there is n^2 algorithm available. This is too much of a difference.
2. Buggy algorithm – When there is n^2 (or even better) algorithm available why the constraint is so low?
Another Important point.
Please do not give such easy time constraints! Only the best algorithm should pass.
Let me give 2 examples,
1. Lights Off – Here n^6 algorithm was passing without optimization when there is n^2 algorithm available. This is too much of a difference.
2. Buggy algorithm – When there is n^2 (or even better) algorithm available why the constraint is so low?
Thank you everyone for your feedback, this is extremely valuable in helping us improve the site. It’s tough to design a contest that is appealing to both beginners and experts but we will try our best.
-Based on everyone’s feedback it seems like having 2 contests, one long (10-15 days) and 1 short (6 hours-2 days) will work for next month. (thanks @Rahul, @Srihari, @Divye, @Neelesh)
– @Ajay, I agree that the problems were not as challenging this time around. It isn’t easy to come up with difficult problem. We will do a team contest in the near future.
– @Tomek We will make it clear what the memory limits are.
-@Michael Is there such a thing as a “new” problem? We do our best to come up with problems that aren’t standard algorithms.
-@Neelesh – some interesting alternatives, we will look into them. We will try to be as clear as possible in the problems, I think what you say about splitting into sections makes sense. We try to be as active and communicative as possible, though we get a lot of email, tweets, comments, forum posts etc… we try to answer as many as we can. I promise it’s not because we don’t care, we just need a better way to determine what are ‘critical’ posts versus stuff that can be answered by the community. Still working on this.
-@xxyyzz, we will be holding a CodeChef Cup still TBD but in the next few months.
-@pruntz – the purpose of the tie-breaker is to determine the winner, it will happen again. It is hard for us to challenge the experienced developers without discouraging new comers. We will try to include more difficult problems in the future while not restricting time limits so that users submitting in other programming languages can still pass.
Thank you everyone for your feedback, this is extremely valuable in helping us improve the site. It’s tough to design a contest that is appealing to both beginners and experts but we will try our best.
-Based on everyone’s feedback it seems like having 2 contests, one long (10-15 days) and 1 short (6 hours-2 days) will work for next month. (thanks @Rahul, @Srihari, @Divye, @Neelesh)
– @Ajay, I agree that the problems were not as challenging this time around. It isn’t easy to come up with difficult problem. We will do a team contest in the near future.
– @Tomek We will make it clear what the memory limits are.
-@Michael Is there such a thing as a “new” problem? We do our best to come up with problems that aren’t standard algorithms.
-@Neelesh – some interesting alternatives, we will look into them. We will try to be as clear as possible in the problems, I think what you say about splitting into sections makes sense. We try to be as active and communicative as possible, though we get a lot of email, tweets, comments, forum posts etc… we try to answer as many as we can. I promise it’s not because we don’t care, we just need a better way to determine what are ‘critical’ posts versus stuff that can be answered by the community. Still working on this.
-@xxyyzz, we will be holding a CodeChef Cup still TBD but in the next few months.
-@pruntz – the purpose of the tie-breaker is to determine the winner, it will happen again. It is hard for us to challenge the experienced developers without discouraging new comers. We will try to include more difficult problems in the future while not restricting time limits so that users submitting in other programming languages can still pass.
@chef The idea is to have at least a few hard problems so that its challenging for experienced people and at the same time have some easy problems so that new comers could try them.
About having problems which are not standard, I dont think you can put ‘lights out’ puzzle in that category. There is a wikipedia page on the problem, where the solution is clearly described, The problem image also is taken from the wikipedia page.
http://en.wikipedia.org/wiki/Lights_Out_(game)
@chef The idea is to have at least a few hard problems so that its challenging for experienced people and at the same time have some easy problems so that new comers could try them.
About having problems which are not standard, I dont think you can put ‘lights out’ puzzle in that category. There is a wikipedia page on the problem, where the solution is clearly described, The problem image also is taken from the wikipedia page.
http://en.wikipedia.org/wiki/Lights_Out_(game)
Prunthaban says:
>I do not understand why we should limit the number
> of submissions when you are going to have rejudging
> too. When you have rejudging, you need not restrict
> submissions.
Suppose some evil person like me submits 3000 submits per day in the next competition. You don’t think that’s too many?
Prunthaban says:
>I do not understand why we should limit the number
> of submissions when you are going to have rejudging
> too. When you have rejudging, you need not restrict
> submissions.
Suppose some evil person like me submits 3000 submits per day in the next competition. You don’t think that’s too many?
I accept that it is not easy to come out with difficult problems. But I think now you have some good people within and outside to set quality problems for you. So hoping to see some good ones in coming months.
I do agree that with such a long format, as CodeChef gains popularity the tie-breaker will decide the winner. But it should not be like “only” the tie-breaker matters.
Frankly, for the last 3 months we had at least one really good problem
April – A5
May – LICS (though it was easy to search on net) and Swappable sequence
June – There were 3 good problems.
This month we did not get anything of that kind. Hope to see more in future.
I accept that it is not easy to come out with difficult problems. But I think now you have some good people within and outside to set quality problems for you. So hoping to see some good ones in coming months.
I do agree that with such a long format, as CodeChef gains popularity the tie-breaker will decide the winner. But it should not be like “only” the tie-breaker matters.
Frankly, for the last 3 months we had at least one really good problem
April – A5
May – LICS (though it was easy to search on net) and Swappable sequence
June – There were 3 good problems.
This month we did not get anything of that kind. Hope to see more in future.
@Tomek
So essentially your point is about preventing spammers (which is a good thing). But CodeChef is trying to use this ‘limiting submission technique’ to prevent people from gaming the system, learn about test-cases, etc. If that was their only purpose, then limiting submission is not the best option.
Coming back to your point, I agree that to prevent spammers we need to limit submissions (but spamming is something yet to happen in CodeChef). But for that, having the limit at around 15 seems very punishing. 150 is a descent number. But there is still a chance that we exhaust them in the first 5 days.
So in-case if we want to implement a limit what about making it a ‘per day’ limit? Even then usually the last few hours are intense fighting and we will be already worrying about how to get a 0.001 increase over others. In addition to that maintaining a count of the submits and worrying about that too will be a pain.
That said, I am completely ok with a reasonably high limit (so that we need not care about it if we are playing ‘fair’).
For example here are some stats from this time.
Balakrishnan – 577 submits
Ashutosh Mehra – 641 submits
So it looks like the maximum we get is around 600. So any limit taking these stats into account (of course these many submissions were done because they ‘know’ they have unlimited chance. So giving a limit they will play more carefully and will not reach such high limits) should be fine with everyone I hope.
@Tomek
So essentially your point is about preventing spammers (which is a good thing). But CodeChef is trying to use this ‘limiting submission technique’ to prevent people from gaming the system, learn about test-cases, etc. If that was their only purpose, then limiting submission is not the best option.
Coming back to your point, I agree that to prevent spammers we need to limit submissions (but spamming is something yet to happen in CodeChef). But for that, having the limit at around 15 seems very punishing. 150 is a descent number. But there is still a chance that we exhaust them in the first 5 days.
So in-case if we want to implement a limit what about making it a ‘per day’ limit? Even then usually the last few hours are intense fighting and we will be already worrying about how to get a 0.001 increase over others. In addition to that maintaining a count of the submits and worrying about that too will be a pain.
That said, I am completely ok with a reasonably high limit (so that we need not care about it if we are playing ‘fair’).
For example here are some stats from this time.
Balakrishnan – 577 submits
Ashutosh Mehra – 641 submits
So it looks like the maximum we get is around 600. So any limit taking these stats into account (of course these many submissions were done because they ‘know’ they have unlimited chance. So giving a limit they will play more carefully and will not reach such high limits) should be fine with everyone I hope.
@pruntz – the purpose is not to prevent spamming, rather to prevent people from figuring out the exact inputs and precompute answers. the suggestions was 15 accepted solutions (though this is an arbitrary number), i don’t think we have reached a proper conclusion on this yet, we hope to get more feedback from the community before announcing the rules for the next contest.
@pruntz – the purpose is not to prevent spamming, rather to prevent people from figuring out the exact inputs and precompute answers. the suggestions was 15 accepted solutions (though this is an arbitrary number), i don’t think we have reached a proper conclusion on this yet, we hope to get more feedback from the community before announcing the rules for the next contest.
Personally, I like putting a cap on submission frequency rather than limiting the number of submissions. Even though they both achieve the same goal, but the former will “prevent” me from exhausting my allowed submissions.
With more and more participants, it may be hard to come up with problems that can hold some of the participants here for 7-8 days. In that case, having a shorter contest may be the only way out so that it does not come down to the tie-breaker. Having said that, I enjoy tie-breaker like problems and would not mind a separate contest of those only (kind of like Topcoder marathons).
Personally, I like putting a cap on submission frequency rather than limiting the number of submissions. Even though they both achieve the same goal, but the former will “prevent” me from exhausting my allowed submissions.
With more and more participants, it may be hard to come up with problems that can hold some of the participants here for 7-8 days. In that case, having a shorter contest may be the only way out so that it does not come down to the tie-breaker. Having said that, I enjoy tie-breaker like problems and would not mind a separate contest of those only (kind of like Topcoder marathons).
I’m just saying. Do these problems seem familiar?
The Quadratic Formula (Mod p)
-http://acm.uva.es/archive/nuevoportal/data/problem.php?p=2123
Fliptile
-http://acm.pku.edu.cn/JudgeOnline/problem?id=3279
I’m just saying. Do these problems seem familiar?
The Quadratic Formula (Mod p)
-http://acm.uva.es/archive/nuevoportal/data/problem.php?p=2123
Fliptile
-http://acm.pku.edu.cn/JudgeOnline/problem?id=3279
If you are planning to have a very-short duration contest, like 6 hours or 8 hours, then there is no need of a single tie-breaker. In a short duration contest you can also score a solution based on how fast the coder submits. In fact I think the main purpose of a short-duration competition is to test the speed of problem solving.
If you are planning to have a very-short duration contest, like 6 hours or 8 hours, then there is no need of a single tie-breaker. In a short duration contest you can also score a solution based on how fast the coder submits. In fact I think the main purpose of a short-duration competition is to test the speed of problem solving.
I suggest for challenge problem , data will be random like we have any Topcoder Marathon Matches . Also the length of problem statement must be limited so that coding contest not turnout english tests.
I suggest for challenge problem , data will be random like we have any Topcoder Marathon Matches . Also the length of problem statement must be limited so that coding contest not turnout english tests.
1. “We need to define rules better. …”
it is not possible for the Admins alone to figure out the crooks. There should be a way for participants to inform the Admin if they suspect anyone. As of now, I don’t know how to inform the Admin regarding such a thing in private. Apologies if the details have been stated and I have overlooked them.
2. Admin responses in forum
You could leave some of the queries to be answered by the community. But there should be upper limit, as in if a query has been unanswered for more than some number of hours, then it must be looked into by the Admin himself without more wait.
3. I love the contests. Keep them going! 🙂
1. “We need to define rules better. …”
it is not possible for the Admins alone to figure out the crooks. There should be a way for participants to inform the Admin if they suspect anyone. As of now, I don’t know how to inform the Admin regarding such a thing in private. Apologies if the details have been stated and I have overlooked them.
2. Admin responses in forum
You could leave some of the queries to be answered by the community. But there should be upper limit, as in if a query has been unanswered for more than some number of hours, then it must be looked into by the Admin himself without more wait.
3. I love the contests. Keep them going! 🙂
My thoughts:
a) I’m happy with a week long contest. While there was room for optimisation on the challenge problem, I basically solved the rest of the problems in the first few days, played around with the challenge for a while, and then just sat back and waited, knowing a heap of people would end up solving the problems and it would come down to who felt like spending the most time playing with minor optimisations for the challenge problem. 32 people solving all problems was probably a bit too high.
b) Another contest in the month is excellent, but whatever you do, PLEASE don’t make it less than 24 hours long – make it a multiple of 24 hours. A worldwide contest is ruined otherwise; I have to put up with the fact that most online contests are in the middle of my night, but I don’t want this to happen to Codechef. Please keep it a true worldwide contest, rather than just for people who are lucky to have it suit their timezone.
c) I agree that forum posts need to be checked more regularly. The description of the challenge problem was completely wrong for the majority of the contest, and I’m sure lead to many people getting frustrated with the fact that any optimisations gave them a worse score.
And finally, d) While this may not be possible with the current online judge system, how about something like this to make challenge problems work better. Provide two sets of test cases; one which works like currently with unlimited submissions, and one, which is used in the final scores, with limited submissions per day which doesn’t display the results for a day (or something similar). That way you can still play with your solution on the first set, still be able to make sure the real case will work fine on your code, yet prevent the reverse engineering or mass submissions issues.
e) It was still a great contest 🙂
My thoughts:
a) I’m happy with a week long contest. While there was room for optimisation on the challenge problem, I basically solved the rest of the problems in the first few days, played around with the challenge for a while, and then just sat back and waited, knowing a heap of people would end up solving the problems and it would come down to who felt like spending the most time playing with minor optimisations for the challenge problem. 32 people solving all problems was probably a bit too high.
b) Another contest in the month is excellent, but whatever you do, PLEASE don’t make it less than 24 hours long – make it a multiple of 24 hours. A worldwide contest is ruined otherwise; I have to put up with the fact that most online contests are in the middle of my night, but I don’t want this to happen to Codechef. Please keep it a true worldwide contest, rather than just for people who are lucky to have it suit their timezone.
c) I agree that forum posts need to be checked more regularly. The description of the challenge problem was completely wrong for the majority of the contest, and I’m sure lead to many people getting frustrated with the fact that any optimisations gave them a worse score.
And finally, d) While this may not be possible with the current online judge system, how about something like this to make challenge problems work better. Provide two sets of test cases; one which works like currently with unlimited submissions, and one, which is used in the final scores, with limited submissions per day which doesn’t display the results for a day (or something similar). That way you can still play with your solution on the first set, still be able to make sure the real case will work fine on your code, yet prevent the reverse engineering or mass submissions issues.
e) It was still a great contest 🙂
Chef : The contest needs to be shorter. Next month’s contest will be one week long. We will also be including a shorter duration contest at the end of next month (a few hours to a weekend – still deciding).Rahul : Well , honestly I wouldn't want the contest to be shorter for I am a novice and I found the longer duration of the competition to be an advantage for me , not just in getting my submissions accepted but in improving my algorithmic skills as well . My general scenario was reading the problem statement and then making a brute force solution thus getting TLE , then trying to optimize it and still get another TLE . Then however I would search Google for new and better algorithms and try to implement them and keep on doing this until I get a solution accepted . During the June Algorithm Challenge , I had to submit nearly 50 submissions per problem to get my solution accepted and at least 10 of them had different strategies and various had different types of optimizations thus explaining and answering some of my questions as to which solutions are better and why . Most of my time during June Algorithm Challenge , in fact all of it , was gone into thinking about “Why did that Algorithm gave an TLE ” and thus learning about new algorithms . Even after the contest , most of my time was spent in learning new algorithms , strategies etc and solving problems at other judges like UVa and I am able to see some improvement in me . No more of my programs are getting TLEs at other judges now . So I would really really wish for a longer contest .
Chef : The contest needs to be shorter. Next month’s contest will be one week long. We will also be including a shorter duration contest at the end of next month (a few hours to a weekend – still deciding).
Rahul : Well , honestly I wouldn't want the contest to be shorter for I am a novice and I found the longer duration of the competition to be an advantage for me , not just in getting my submissions accepted but in improving my algorithmic skills as well . My general scenario was reading the problem statement and then making a brute force solution thus getting TLE , then trying to optimize it and still get another TLE . Then however I would search Google for new and better algorithms and try to implement them and keep on doing this until I get a solution accepted . During the June Algorithm Challenge , I had to submit nearly 50 submissions per problem to get my solution accepted and at least 10 of them had different strategies and various had different types of optimizations thus explaining and answering some of my questions as to which solutions are better and why . Most of my time during June Algorithm Challenge , in fact all of it , was gone into thinking about “Why did that Algorithm gave an TLE ” and thus learning about new algorithms . Even after the contest , most of my time was spent in learning new algorithms , strategies etc and solving problems at other judges like UVa and I am able to see some improvement in me . No more of my programs are getting TLEs at other judges now . So I would really really wish for a longer contest .