We started the new year with the hope of making the first Cook-Off of this year as the most memorable one. And we did ensure that we get the maximum crowd in for the celebrations. Anudeep Nekkanti was handed over the responsibility of creating the problem set. Who else could have done the job justice? Lalit Kundu, Minako Kojima and Sergey Kulik completed the team.
Not for the reasons that we wanted, the story of January Cook-Off 2015 will still be remembered. For the load that the server could not handle and crashed, many times! This will be the first ever unrated Cook-Off at CodeChef.
The problems of Anudeep Nekannti were much appreciated. To add to that, the problem statements were named after the members of our team at CodeChef. We take this opportunity to express our gratitude to Anudeep for this gesture and an awesome problem set. We feel humbled and sad to have let this problem set down.
When a contest goes wrong, it is you, the users, who suffer. We as organisers share the complete responsibility for this to have happened and apologise to all of you who took part in the contest. We goofed up once again. The biggest sufferers are the problem author and the panel who spend a huge amount of time and effort to bring the problem set to you. We apologise to Anudeep and the entire problem setting panel, including Lalit, Minako, and Sergey for not giving them the steady two and half hours that they needed to have for this deserving problem set. We are sorry.
To be fair to all the participants who put in all their heart and soul trying to access the problems, we are making this Cook-Off unrated. Yes, you will not be rated for the contest and there will not be any winners. And we leave it there.
Now, let us explain on what went down in those seemingly never-ending two and half hours and what have we done to ensure that nothing like that ever crosses our way again.
What was the “Cause”?
We had just launched a new feature to show the accuracy and the count of solutions to “subtask” type problems on the contest page separately. This made us fire some expensive queries, which shot up the load on MySQL. Some indexes were missing and MySQL was hogging more CPU. We did not do any profiling after the launch of this feature. The dogpile effect that occurs at the start if any contest and something that we fought hard to reduce game back in a big way and never let the MySQL server recover. It ensued the site crashed. And the rest you know.
What we did to address it then?
It took us a while to identify this. As soon as we did that, we took down the feature, which although made the site run normally, took a lot more time than we would have wanted. And way more time for the contest to be fair to anyone. We also fix the expensive queries in the mean time. However, it was only a temporary fix and we went straight to the drawing board to find a permanent fix to the problem.
Do you have a permanent fix now?
Well, we have found a fix, now whether it is permanent or not, we will find that out in the upcoming Cook-Offs and other high octane contests in 2015. By the time of writing this blog, we have run many contests including the February Cook-Off, and things seem to be under control. However, to be frank, we have not yet received as much load as we did in the January Cook-Off.
What is the fix?
To start with, we have profiled and fixed all the misbehaving queries to make sure that they do not cause any such trouble. While investigating the cause, we also found that our rate limiter was put off, which contributed significantly to increase the load. We have enabled it again. Also, we took this opportunity to dig hard and identify what can be better fixes to reduce the dog pile effect at the start of our contests and improved the way we were creating our cache as well as locking the cache. So far this has worked and we believe the issue has been fixed.
We know, this is not the first time we have “goofed up” in a Cook-Off and we have tried hard to get things fixed. So, do not get dismayed and keep the problem cracking going. Also, keep sending your love for the chefs at CodeChef. We do need that.
That will be all for our unrated January Cook-Off 2015. We hope it answers your queries regarding the issue during the contest, the ratings or the winnings of the contest. However, if any of you still have it, feel free to write to us at: firstname.lastname@example.org
Till next time, TA-TA
See you at the contests.