I'm running the ICFP programming contest
13 points by abhin4v
13 points by abhin4v
Every year I forget that ICFP is coming up & fail to put a weekend aside to have fun programming.
Clearly I should be putting it in the diary a year ahead...
Reminds me of ORCA! https://100r.co/site/orca.html Loving the concept
The grid-based editor we built for the language is directly inspired by our friend River's editor for CLAVIER-36, which was originally an offshoot of ORCA!
Looks fun. But what actually prevents you from having an LLM read the textbook and problem descriptions?
Claude, please take a long look in the mirror for me and figure out what kind of person I've become.
(I'm the lead designer for this year's contest)
I'm not being snarky at all here - try pointing an LLM at the textbook and asking it about how the contest design interacts with current model capabilities! I've found models to be pretty good at evaluating their own strengths and weaknesses in this domain; we did a lot of testing like this while developing the contest.
Concretely, nothing is stopping you from having an LLM read the textbook and problem descriptions. Our rules for this year explicitly permit AI usage.
Love the programming language, by the way. I wrote a lot of code in 2D esolangs a decade ago, if I had seen this (and were free) I would have definitely tried it out.
Ah hey thanks!
We're planning to turn the contest into a slightly more approachable game (a lot of the problems are pretty hardcore right now) and get it back online at some point. We have definitely not figured out the details yet (we're busy with the contest, our infra will need to change, etc) so it'll be a bit, but you'll definitely have a second shot (not limited to a weekend) in the future.
Honesty and/or a desire to have fun programming by oneself?
I guess I'm glad my entries were in the early to mid 2000s!
Captain of a team placing:
2001: 2nd
2003: Judge's prize
2005: 2nd AND Judge's Prize
(Note: I am completely unaffiliated with this project, I just read the description.)
Clearly, one can point point a top-tier LLM at any problem and expect it to do a job comparable to a low to middling skill programmer. But I do think that this particular programming language is specifically designed to be resistant to LLM success.
One of the fundamental limitations that LLMs have is that they produce their output sequentially. They spit out a series of characters (or tokens) one after the other.
This language is designed to require two-dimensional thinking. I would find it difficult to believe that any human could sit down and write out a non-trivial "program" in this language working left to right, top to bottom. The effective way to work is to take a big grid and start filling it in, making errors and corrections as one goes along. But this is something that an LLM cannot do naturally.
Sure, I could build a tool to consume some other representation and spit out executable code in this language (although, it would be somewhat tricky to do so). And I'm sure someone seriously dedicated to making an LLM write programs in this language, could get an LLM to build such a tool. But simply showing the language definition and problem statement to an LLM is not going to be particularly effective.
Important note: this is me just theorizing based on general principles; I have not tried it.
The game designer did write this:
Of course, we spent a lot of time thinking about how to design a fun programming contest in the age of AI agents. It would not be satisfying if the winning strategy for the contest was to run an unmonitored agent in a loop for 72 hours.
That is exactly what my original comment was responding to.
I know they said that. I'm just not sure how that can be achieved, or whether it has been.
The comment you replied to offers a theory.