Hosting fastest code challenges
When posting a fastest-code challenge, what do I need to bear in mind?
Unlike with code-golf challenges, where the score is simply the number of bytes in the source code, for a fastest-code challenge the time taken to run will be different on different machines. What options are there for defining an objective way to measure the score of an submission? Does this necessarily involve the challenge host running all of the submissions on their own machine?
How can I ensure the challenge is suitable for as many programming languages as possible (not excluding languages which are unlikely to top the leaderboard but where optimising might still be an interesting problem - competing against other submissions in the same language)?
1 answer
If restricting everyone to the same system, then we can force every contestant to use https://tio.run and the on-site bench-marking found below "debug" -> "real time" as efficiency metric. Example.
Advantages:
- No matter how (in)accurate, this gives everyone the same conditions and the same bench-marking system. It is fair.
- Multiple programming languages and compilers are possible. Competitions between solutions in the same language and/or other languages is possible.
- We already use TIO for most challenges.
Disadvantages:
- Heavily biased towards a specific target - I think it runs on x86_64 Linux?
- Execution times are rather arbitrary with poor accuracy. This is a poor benchmarking tool by professional standards, but maybe for this purpose we don't really care?
1 comment thread
1 comment thread