I’m a former CTO. Here is the 15 sec coding test I used to instantly filter out 50% of unqualified applicants
71 points by WilhelmVonWeiner
71 points by WilhelmVonWeiner
This is a dumb and illegal test.
tl; dr: He shows candidates a 6-line Python code snippet and asks them the result, but the code snippet has a = operator that doesn't appear visually but is included when you copy/paste the code snippet (it has an inline style attribute of opacity: 0; font-size: 1px). So, the visually rendered code produces one result and the actual code produces another. The author considers the visually rendered code to be correct and the actual code to be incorrect.
In the US, this interview question would be illegal, as it causes non-sighted readers to "fail." Author's company was based in Canada, and I don't know accessibility laws there, but I'm guessing illegal there as well.
This reminds me of a joke: When hiring, we immediatly throw half the applications in the trash - we don't want to hire unlucky candidates.
It seems like a joke but there's the Secretary problem in math. One of the optimal strategies is to review but automatically reject the 37% first ones. And then choose the best one compared to the ones you automatically rejected.
The joke is that you throw the "unlucky" half in the trash immediately, without even looking at them (because they are unlucky). In the secretary problem, you interview and then reject the first 37% candidates, and then use the best (rejected) candidate so far as a benchmark.
Entirely different things.
The secretary problem is neat, but there is a constraint that makes it not applicable to most hiring processes I have been involved in at least: A decision about each particular applicant is to be made immediately after the interview. Once rejected, an applicant cannot be recalled.
Maybe I'm understanding you backwards, but the "immediate yes/no" constraint is precisely when the "reject first 37%" solution becomes interesting. If you can give your answer after having reviewed all candidates, there's no mathematical conundrum.
I think they're arguing that the secretary problem doesn't apply to real life hiring processes, since real life hiring processes don't have the immediate yes/no constraint.
Am I reading the same article as you? I thought that's how the secretary problem works too?
Yes, the constraint is in the secretary problem definition, but in my experience it is not a common constraint in real world hiring. Instead companies just review all applications and give their decision when they can chose the best candidate from all the candidates.
Ahh, you mean that way around.
When I've been hiring, good candidates have been far enough apart that they'll have another offer if I wait to compare them.
Try discussing hiring in these terms with your recruiters and come back to me with the results.
I thought this was a standard hiring practice at some of the more "hire 'em young, break 'em down, burn 'em out" type of consulting firms - and while it is darkly funny, it's also reflective of how much they actually care about employees.
Ironically, this is just above the “You Can’t Opt Out of Accessibility” (https://lobste.rs/s/rqub7y/you_can_t_opt_out_accessibility) story right now.
It's ironic because someone with a screen reader would be fine because of the aria-hidden="true".
That ARIA attribute wasn't there in the original. See: https://lobste.rs/s/zzvn7y/i_m_former_cto_here_is_15_sec_coding_test_i#c_1ie4xy
I'm not sure if it would be illegal if you just ask first if you need adaptations for vision. It is common in high school or university exams to do a different version to accommodate for this kind of stuff.
You're saying it's not illegal if you don't ask visually impaired applicants this question, but they didn't say anything about pre-screening for accessibility, so I think it's safe to assume they asked everyone and screened out all visually impaired applicants without knowing it.
In the discussion on LinkedIn, somebody suggests adding ARIA attributes for screen readers, and the author replied that it was "a nice improvement" so it doesn't sound like they previously considered accessibility.
Aside from that, this post just sounds ridiculous.
Someone got rejected, then applied again with a different answer even though they didn't know they got rejected for their answer? How often do you reapply for the same role immediately after getting rejected?
47% of candidates got the right answer? In what world? Like 10% of applicants can even do fizzbuzz correctly. Who are these 47% of unemployed programmers who are so fast at running a for loop in their head that they don't do the sensible thing of just running the code in an interpreter?
You'd really run that in an interpreter instead of treating it like a crossword or brain-teaser for a little fun? It's really not that deep.
Yeah, if the penalty for getting it wrong is wasting 15-90 minutes on a job application, I'd definitely run it in an interpreter because it's definitely going to get the right answer, whereas my brain might make a mistake.
Similarly, I use a calculator/spreadsheet formulas when I do my taxes even though I could do most arithmetic in my head, but the penalty for doing it incorrectly is high and my brain isn't as accurate as a calculator.
The question is there to find out if you're confident in your reasoning skills to wing it or if you'd palm it off to the interpreter/LLM, so I guess in our cases it works out fine?
The bias here is assuming that you view it as low stakes.
I worked through it in my head because I was reading a blog post and there were no negative consequences of getting it wrong. If I’d been applying to a job I really wanted, or was in a situation where I really needed a job, I would have probably tried running it. Given enough time, I would have probable have checked why my answer didn’t match the one executing it gave and seen the equals that showed up in the pasted version.
In short, in a high-stakes environment I would not rely on my interpretation when I could mechanically check my understanding.
So this is going to filter out people who really need or want the job.
There are two kinds of hiring processes: ones that have known bias and try to compensate for it and ones that have unknown bias. This is the latter kind.
I've learned the hard way that you are not me, and what I might consider obvious might not be obvious to you. People are going to approach the code in different ways, and in ways that make sense to them, but not to you, or even to me.
The rest of the article, subversion, and everything else aside...
Taken at face value, if this snippet is not obvious to someone in a matter of seconds, there is no way that they're an experienced (good) software engineer. It's simply not possible. There is no way that they could ever get anything done on their own without a very simple, very fast mental model for something this trivial.
It is obvious to me that it is a for-loop with three iterations, and a variable that is modified each iteration, in one of two possible ways depending on the value of the iterator. I have been bad at doing math in my head my entire life and if we to this add to this the stress of this being an interview setting I definitely would become a deer in headlights when it comes to actually getting to what the final value is. I refuse to believe that the difference between hardware and software engineering is what explains my professional success.
I have been bad at doing math in my head
I think there's a difference between "bad at doing math in my head" and "can't do simple arithmetic with single-digit numbers"
Perhaps I should have used different phrasing, or additional phrases. There issue is with correctly translating the program into the kind of state machine I can run on my brain and then doing so accurately and with confidence in a stressful context over several clock cycles.
This is however based on my interpretation that I'm being put on the spot in real-time. There are enough hints to suggest that ithe test is over mail and over some longer time period, in which case my action would definitely be to copy it to a python prompt and run it, because they're is absolutely nothing interesting about the code or the question.
I use “reader view” on many web sites to increase legibility. I would have failed this test, because the hidden = shows up in reader view.
This test is garbage.
I don't disagree that tbe test is garbage, however I don't think the existence of false negatives by itself is a problem. Every test will have false negatives
It makes sense that you’re using reader mode for an article, but it would presumably not be available on a page that’s more like a web form for your job application.
This also filters out people with higher-order thinking skills!
What's the chance that when I try to run this code in my head I make some silly mistake? Low, but certainly possible. Especially since it's an interview question and is likely to have a gotcha in it.
What's the chance that the Python interpreter gets it wrong? Basically zero.
So someone who can reason about their own fallibility is likely to copy-paste the code.
Given this question myself, I would have run it both in my head and in Python, noticed the discrepancy and emailed them to ask what was going on. Seeing how the CTO reacted to being told about the downsides of their clever interview question would be informative!
A solution could be to add a hidden comment to inform blind users
But how to hide that comment from LLMs?
Or indeed sighted users when copy-pasting? (A comment magically appearing would stand out more than a subtle change to greater than or equal)
Cool that the first thing AI is doing is shattering the trust that was the basis of accessible equal opportunity.
It's just so expensive to be naively trusting in an era when there's so many people who would take your trust and abuse it in a selfish way. A trusting policy is unlikely to even help the people you want to help when it would bring in a million ethics-free opportunists who care not at all for how much of your time they waste so long as it's cheap to do. I think ruling out the huge swarm of AI imposters might actually be a significant boost for a talented candidate who also has a disability.
This is nothing to do with ai. Before ai you could paste to repl. And after ai you can proof what was pasted
In the US, this interview question would be illegal, as it causes non-sighted readers to "fail." Author's company was based in Canada, and I don't know accessibility laws there, but I'm guessing illegal there as well.
I really don't know about US laws, but couldn't they have a specific test for visually impaired people? I remember that every time I applied to an US company I needed to fulfill a form saying any kind of disability, so I assume this was one of the reasons why this was asked.
screen readers would show the correct text because of aria-hidden="true"
It is kind of silly and dumb, but it's a big jump to say it's illegal. If they discriminated against blind applicants using the question, that would be illegal, of course, but nothing in the article says they do that.
The article says they auto rejected any candidate who gave them the "wrong" answer, but any candidate using a screen reader would give the "wrong" answer if they solve it correctly. That illegally screens out applicants who use a screen reader.
You don't have to intentionally discriminate to break the law. If I create an ML model that rejects 100% of black candidates, even if I didn't train it with that goal, it's still illegal to use that model for hiring decisions.
The article says they auto rejected any candidate who gave them the "wrong" answer,
Where does it say that?
I see the oppposite:
A few candidates resubmitted the application after getting the answer wrong (we didn’t tell them), one of those candidates was a great hire.
The article says they auto rejected any candidate who gave them the "wrong" answer,
Where does it say that?
The entire article says that. It's in the title and several lines in the body:
Here is the 15 sec coding test I used to instantly filter out 50% of unqualified applicants.
Add a programming knockout question to the application process
How are you interpreting the article in a way that means they didn't reject candidates with the wrong answer?
You quoted the part where they said that candidates resubmitted the application, meaning they had to reapply after the author rejected their original application for having the incorrect answer. Once they fixed the answer, they considered them for employment and supposedly ended up hiring one.
If I auto-reject you once and then decide to hire you, does that change the fact that I auto-rejected you? More importantly, does it change the fact that I auto-rejected an unknown number of disabled people who didn't bother to reapply?
Your assumption is very incorrect. I just tested it with iOS’s VoiceOver, and it reads the modified line as “x greater than 3”.
Looking at the page source, it marks the hidden “=” as aria-hidden="true" making it invisible to screen readers.
It looks like the author foresaw this issue and took deliberate steps to resolve it. (Unless it was edited in response to recent comments like yours?)
There's already a fair bit of outrage at this building in the comments, but I take this as a different kind of warning - this is reflective of attitudes in the C suite in many companies: they come up with an idea and believe it's a good one, but fail to evaluate or understand the impact. Ultimately, this is one of the types of personalities that people work for.
This is exactly the reason diversity is valuable intrinsically. This "test" -- and this blog post -- wouldn't have gone out if the author had a friend with visual accessibility needs.
Note on aria-hidden="true": While not a CSS style, it works with these styles to tell Screen Readers to ignore this element entirely, ensuring the "invisible" character doesn't confuse users relying on assistive technology.
We had an open question: "Describe the most interesting issue you ever debugged." After some responses, the LLM answers were very easy to pick out: they were vague and high-level, made sure to stress why the slop was an answer about debugging and that it somehow impacted revenue, and they were almost always variations of the same vague scenario ("intermittent connectivity issues"). Just seeing the word "intermittent" was already a 90% indicator that it was AI slop, though I always read the full response to give genuine candidates the benefit of the doubt. Conversely, for the answers written by humans, they contained a lot of signal about whether the candidate would be a good fit for the role.
Oh, I have a fun story with that though! I had a program that uploaded user submissions to .... I think it was amazon s3 but maybe a similar provider (this was back in 2010ish so been a while) and it had..... intermittent connectivity issues :P People would randomly complain the upload failed but it worked pretty consistently for me. But it failed about 1.5% of the time with an "unauthorized" error.
Well, hint time: to upload the file to the provider you had to couple it with a hash after padding to length a multiple 64 bytes. Guess what 1/64 is? It was an off by one error in the padding impl - if it was already a multiple of 64, it added a byte anyway, correct in other cases. The unittests looked so comprehensive and claimed 100% coverage... but never actually covered that edge case correctly.
Fun story, still one of my favorites to tell even 15+ years later, a couple good lessons there.
Yes, another benefit of this question was that once in a while, I got to read a really good story :D
Be careful. Filtering out the use of words like "intermittent" will filter out three sorts of candidates:
Admittedly, you do want to filter our two of those three groups, but the third has some good candidates. The AI got the habit of using words like "intermittently" because they're often used in incident reports/technical write-ups from people doing valuable work.
By the end of high school my teacher was basically drilling us at Cambridge CPE level, which was all about exceptions, transformations and fancy phrases. It was also super stressful. In a high pressure situation, like an interview, these drills still kick in and I write using school patterns, or phrases I picked up from literature. I hope it's too clumsy to be mistaken for LLM output, though.
And yes, I very often use word "intermittent". And I would definitely use it when answering that question, because intermittent issues are elusive. And interesting to debug. I love debugging interesting issues. I also like em–dashes!
And if somebody rejects my application because of that, great! I wouldn't want to work with somebody who does that.
You're right, I left you out by accident. Your comment reminds me that I even read a great post about this from a Kenyan writer yesterday or the day before, about the frustration of being mistaken for AI.
https://marcusolang.substack.com/p/im-kenyan-i-dont-write-like-chatgpt.
I sincerely believe programming is one of the most fulfilling activities I can ever take part in, yet I cannot think of a fulfilling answer for this question (certainly not one that would paint me as a good hire for the interviewer). I barely remember any issues I worked on, and those I do remember have always been small things. E.g. typos messing up runtime behavior in an unexpected way, skewed (or assumed!) Interpretations of system interfaces.
Asking for a friend: for juniors with a similar experience, would you consider this a serious problem? What would be a good way for the interviewee to "repair" a genuine but boring answer to this question?
Edit: I finally thought of a good and semi-recent example, but it's not one I thought was particularly fun. A highlighter (Pygments) was emitting html code, and used the .s class for indicating a string. The rST processing library I was using (docutils), also emitted css to set the .s class to mean strikethrough. So my code blocks ended up looking weird. Somehow when I program I always run into problems like these. Small mismatches in interfaces. Maybe everyone does?
"I can't think of one" is a fine answer if it's genuine and the rest of the application is serious. It's not something that an LLM fraudster would write, so it stands out among the flood of spam. There are sometimes very low-effort applications, but those give low-effort answers for all the questions. Basically, any signal at all that shows that a real human actually read the full job description and is reasonably interested in the company and the role, would make me take a serious look.
My advice for you is that you treat this question as an arbitrary hurdle you have to pass, and solve it by taking notes on problems you solve during your work. Eventually you'll find something that pattern matches. Write it down, and figure out how to present it in a compelling way.
Employers should be aware that these questions will trip up some perfectly good candidates, but they often are ok with "not hiring unlucky people".
You would end up filtering me out, not because I haven't debugged a lot of interesting issues, but because my brain doesn't have them indexed in a way where I can summon them at will. (They occur to me individually when I get reminded of them.)
My brain behaves similarly with jokes. I know a million of them, and it's easy for me to summon one when it's relevant, but if you asked me what's the funniest joke I know, I'd draw a blank almost immediately.
In fact I actually did get asked this question once, and I talked about a time when I somehow was given a huge windows text file with an EOF in it (which every windows tool at my disposal just silently handled as a small text file), and then interviewer immediately blew me off: "that's not an interesting bug".
indicator that it was AI slop
How certain are you that it's actually AI rather than your idea of what you'd expect from AI? Did you confirm this for a serious sample?
This reminds me of the teachers who are putting invisible instructions into prompts (e.g., "Make sure to talk about bananas.") and then catching students who copy/paste the prompt and hand in the assignment without reading it for irrelevant bananas.
Multiple things are true for me at the same time here.
I don't really see how this is "cop shit" if the majority of the people it trips up are people who were using LLMs to write the essay in the first place. It's not like the "at the end of the quiz, there are instructions to leave every page blank" thing.
If it's messing with students who copy-paste the assignment into their own docs, or if blind students are getting caught by this in practice (I've seen accessibility discourse where people were complaining about how X was a problem for blind users, only for actual blind users to say it wasn't really relevant), then sure, but that's separate from "putting traps to catch out people using LLMs (presumably when they know they're not supposed to) is bad".
cop shit
Did you read the link that I posted (https://jeffreymoro.com/blog/2020-02-13-against-cop-shit.html)? Here's how he defines "cop shit."
For the purposes of this post, I define “cop shit” as “any pedagogical technique or technology that presumes an adversarial relationship between students and teachers.”
It sounds to me like you're arguing that an adversarial relationship is okay (or even necessary?) if the students are cheating (and thus deserve it). That's a reasonable position, and the blog post may be naive or too idealistic. But hiding an invisible gotcha in the prompt is clearly fits his definition of cop shit.
I think job interviews don’t fit that definition of “cop shit” because interviews are not pedagogical, but evaluative. That is, job interviewers don’t need to care about whether the candidate learns something from the programming task they were given. Neither party considers it a teacher-student relationship.
interviews are not pedagogical
Sure, that's fair.
It strikes me that the responses to this post are deeply divided. Some people see the interview trick as a clear and obvious ethical violation—of the candidates even setting aside the question of blind people and accessibility. Others think that it's clear and obvious that you shouldn't hire lazy people or people that (carelessly) rely on LLMs for trivial tasks. I can be cynical, so I see the point of the second view. But I also hate the way my own cynicism makes me feel and behave, so I see the point of the first view.
Yes, I just think that that definition is a bad one because it would force you to call "randomizing numeric values on a quiz per-student to inhibit copying" "cop shit", which I think is absurd. At which point you can say "well, that's not cop shit because it's fine", but then you have an "it's not cop shit if I like it" carveout at which point it kind of loses its usefulness as an analytical tool.
I still think you may be missing the point of the blog post—or you may just disagree with him very deeply. In any case, he has no interest in methods like that to prevent copying nor in anything like them.
Cop shit is seductive. It makes metrics transparent. It allows for the clear progress toward learning objectives. (“Badges” are cop shit, by the way.) It also subsumes education within a market logic. “Here,” cop shit says, “you will learn how to do this thing. We will know you learned it by the acquisition of this gold star. But in order for me to award you this gold star, I must parse you, sense you, track you, collect you, and—” here’s the key, “I will presume that you will attempt to flout me at every turn. We are both scamming each other, you and I, and I intend to win.” When a classroom becomes adversarial, of course, as cop shit presumes, then there must be a clear winner and loser. The student’s education then becomes not a victory for their own self-improvement or -enrichment, but rather that the teacher conquered the student’s presumed inherent laziness, shiftiness, etc. to instill some kernel of a lesson.
He does not need any carveouts because he is operating with a fundamentally different mindset than you are. Again, to be clear, this doesn't mean that you are wrong or he is right. But, also again, I don't think you have read the post. So we are just talking past one another.
Yeah, I'm sure that gaslighting the people you interview is a great way to start a mutually beneficial relationship.
I think the puzzle is kind of neat aside from being ableist. I've been in many scam interviews (as the interviewer) and having a filter beyond a resume or something they can ask AI about before I'm wasting my time face to face would be great.
Sorry to be this categorical, but intentionally lying to the people you interact with is bad, not neat.
Call it lying, call it a filter. I've had enough of my time wasted by bad faith applicants which is why people like the author create these puzzles in the first place.
Your comment just comes across as kind of naive to me like you haven't had these interviewing experiences?
I disagree, I would not call it naive. be the change you want to see in the world. don't perpetuate lying to candidates... it's not honest. We need more honesty in the world. While I have experienced hiring empty shirts before, it's just par for the course.
While I have experienced hiring empty shirts before, it's just par for the course.
Hiring the wrong person is far more destructive than hiring the right, or even a mediocre person. Especially in corporate environments where it can take a full year to fire someone unless they're actively engaged in illegal acts in the workplace.
intentionally lying to the people you interact with is bad, not neat
Congratulations! You are a human, not a C-Suiter.
Do you really mean gaslighting? If so maybe you don't mean interview? By the author's description of the process the people interviewed are specifically the candidates who answered based on their own perceptions and had those perceptions confirmed rather than denied by the company.
In my interpretation the "true" solution of the puzzle is the one where you run the copy pasted code in a python interpreter. Even for simple cases like these, as a developer I trust the way the machine runs the code more than I trust my interpretation of how the machine will interpret the code. So, when I work from a different version of the code than the one which would be canonically correct, then yes, I would call that gaslighting on the part of the interviewer.
Gotchas are rarely a good indicator. Let alone one that is potentially discriminating based on method of display.
I ran it in my head because 1) it was trivial, 2) I don't use python / have it installed. I guess this may also filter for people who do not have python installed, which may be undesired.
That said, contrary to the other comments, I liked it. :D Maybe those who dislike it, tell us if you have ever been screening candidates. I have, so I can relate to the author's frustrations. (I understand the opposite arguments as well, and they are valid too. Well, hiring is tough on both sides.)
Yeah, I kind of liked the idea, too.
Obviously it couldn't be used to interview people using a screen reader (as so many people are complaining about), but it seems like an okay way to test if people are understanding the code or copy/pasting it somewhere to be evaluated.
It's amusing to me that so many people are upset that the question potentially discriminates against the blind, but few seem to care that so many sighted applicants can't understand 8 lines of code without running it.
I also kinda liked it ... Actually my first reading was wrong: I instinctively said "5" without stepping through it.
It's more like "synthetic code" -- NOT a common pattern. So my brain tried to "compress" it into a pattern that didn't exist.
But 5 wasn't one of the answers, so then I really stepped through it, and got it.
I do think most programmers should be able to step through a 3-iteration loop with an if statement, without access to a computer. And there are probably some that find that too "effortful".
I can understand why people are annoyed by the hidden >= trick. Pre-LLM, I think you would not need that. But I can see the rationale for it now.
I consider that a separate issue from the test itself -- I think this would have landed better if the author had just omitted the "cheat-catcher" trick.
I can also see being annoyed if you're not a Python programmer. I don't really know say Rust or Erlang, so I would likely do worse there.
Obviously, this should not be the ONLY test you give people. But I would say there is SOME signal there.
And there are a lot of interview practices with almost zero signal.
I don't really know say Rust or Erlang, so I would likely do worse there.
It comes out basically the same.
let mut result = 0;
for x in [3, 3, 5] {
if x > 3 {
result -= 3;
} else {
result += 3;
}
}
I liked the idea, and I've also screened candidates (tho it was pre-LLM age). Basically a LLM-age programming captcha.
The accessibility angle is very valid. I do wonder if the proper framing would help:
"Solve the following problem without a search engine, IDE or AI assistance. If you are using a screen-reader or other assistive device please use <em>this page to view the problem</em>."
Anyone spamming job applications w LLMs would never notice or follow the alternate directions, and those who did would get the actual problem.
That framing would be much better. I think the one thing it’s missing is an instruction to avoid using python on your computer, which is not an IDE. An improved first sentence: “Solve the following problem in your head, without a search engine, Python runtime, IDE, or AI assistance.”
With that phrasing, a candidate who copies and pastes the code into python can legitimately be considered a cheater and thus not worth hiring. Without that phrasing, the candidate could have been an experienced Python programmer who just happened to have a Python REPL open in their terminal and wanted to save time or avoid mental mistakes.
The implementation could have been improved to reduce chances of misleading screen reader users (e.g. speak:none, aria hiding or explaining the catch, adding the extra char only when detecting text selection and a copy shortcut).
It shouldn't be an automatic rejection either, without giving candidates a chance to explain why they gave such answer.
But I totally understand rejecting candidates for mindlessly pasting code into LLMs and mindlessly copying the answer. Any fool can do that.
Even if we accept that LLMs are now a part of programming workflow, the new required skill is knowing that LLMs are a wrong tool for computing exact answers.
They were also trying to reject people who mindlessly pasted it into a python interpreter.
For such trivial code I would expect programmers to be able to evaluate the code in their mind quicker than it takes to paste it into a Python interpreter.
I wouldn't immediately reject for a wrong answer, because a follow-up answer could be something reasonable "I knew the right answer, but I feared it's a trick question, so I used the interpreter", but OTOH if a candidate indicates it's too much to bother and they can't engage their brain for 6 lines of code, that's a red flag.
My expectation matches yours. I was just pointing out that it was a broader rejection than pasting into an LLM, with the subtext that I have a lot more appreciation for jumping to a REPL than I do for jumping to an LLM. IOW, I expect jumping to a REPL to be a generally good strategy, even if I don't think anyone on my team should need that for this code. It's not fraught quite the same way as jumping to an LLM.
I’m going insane, the source code does mark the = character as aria-hidden. Was that only added after comments like yours, or did you (and the guy above complaining about accessibility) not check?
It is funny to me because one of the #1 things I tell to people on the internet asking me questions is "what did the compiler say?"
So often, people are like "Can I do X?" and I'm just like.... don't ask me, just try it! Once you have an answer from the compiler, then you can ask me to elaborate on why it accepted/rejected it, but like.... try it yourself. Try a few variants and see if a pattern jumps out.
I think some of the people are asking for an explanation, but I know the majority of them haven't actually tried the code. I suspect some of this is that the default workflow for trying code among the kids nowadays (shakes first at cloud) is awfully heavy - create a "project" instead of just dumping in a temp file and stuff - but still, I'd rather have someone who copy/pastes it to try it than someone who doesn't.
Even if you read it by eyeball, you should probably still confirm your result before proceeding. Another peeve of mine is someone coming in saying "why does this error?" when their pasted sample works just fine because the error was a typo or something they fixed while typing up the reduction. at least paste in the error message so i can guess at that lol.
Though I know I'm rambling, but one more thing came to mind, many years ago someone posted a code sample to a mailing list thread about a Unicode argument. Person A was like "just use code points" and this rebuttal was like "accented_e.length == 1? or 2?" and people actually were copy/pasting it and being like "works for me it is 1". But if you actually downloaded the attached file it would be 2 because he specifically saved the file with inconsistent unicode normalization and that was the whole point of the demonstration. So trying it shouldn't be the end of the investigation either, gotta consider the context of the discussion that maybe there is intentionally a hidden meaning.
So in that direction, I could see value in this test if the interviewer used the different response as a prompt to see what the candidate's next step is when faced with a discrepancy. How many times have you been debugging something, wondering why the hell the code isn't doing what it ought to be doing, then realized you were working on the wrong copy of the file or something like that?
The logic of course is that for a good programmer it would be more of a hassle to copy, open an interpreter or ChatGPT, paste it, run it, then answer, than just run the code in their head.
Really? I just tried it and it took me 10 seconds to copy it into a file and run it with python.
Not to mention the entire article is monumentally dumb. The whole question is based on the idea that what you see is not what you'll paste. You see < in the condition but when you paste you'll paste <=. They do this with CSS trickery! Then there is the hidden expectation that the candidate is supposed to answer the question based on what you see. So a candidate that pastes and executes the code will be rejected. How is this not dumb? Why must someone who cares enough about the correct answer to paste and execute the code be penalized? Arguably, they're the one giving you the correct answer!
And what about blind users? Blind users are going to see not "what you see" but what you paste, so they'll see <= . So this test will fail all blind candidates 100%. How is this even legal?
Really? I just tried it and it took me 10 seconds to copy it into a file and run it with python.
This is the part of this screening question I find interesting and useful in a slightly different form. Not the timing per se but the instinctive responses we have to the author's assertion. I think we, as an industry, have two legitimate dispositions re: computers (stated intentionally imprecisely):
The question as posed doesn't directly ask this but I think our first assessment of code is often biased by our disposition on this question. You likely aren't making a time calculation or an effort calculation when you choose to copy/paste versus read and interpret. For real code you're likely going to do both, so what you do first isn't a sign of your skill but more of how you see your role relative to the computer 's role.
I think this is what the author is really measuring. IME some of the people I've seen dislike their work the most were in a place where the vast majority of their coworkers had the opposite disposition on this question. So I think it really is worth discussing this in interviews. I'm much less confident that it is a good screening tool though.
It’s not even filtering out 50% of unqualified candidates. It’s just filtering out 50% of all candidates.
Also, they hired someone who got it wrong.
¯\_(ツ)_/¯
Ugh, lieing to potential hires is a great filter, well done, that's how you nurture good company culture.
The candidates are lying about having read and understood the code. Disqualifying lazy cheaters should be good for the company culture.
Only the ones for which the filter is successful. Not the many other scenarios outlined in other comments here where an honest (and potentially great) candidate is caught in an imperfect filter.
lieing to potential hires is a great filter, well done
*lying
Put your outrage on hold for a moment here: What is the "lie" in this example?
Personally, if I ever had to interview for a position, I'd hate to be competing with lazy/dishonest/ignorant people who are misrepresenting (i.e. "lying" about) their own abilities by cutting & pasting into Chet Jippity's web site.
The logic of course is that for a good programmer it would be more of a hassle to copy, open an interpreter or ChatGPT, paste it, run it, then answer, than just run the code in their head.
It took me 8.7 seconds to open python, paste it, and type result to get the answer. It took me 12 seconds to do it in my head, and I'm sure it would've taken me double the time in an interview setting. I don't think I'm incompetent at all.
The logic of course is that for a good programmer it would be more of a hassle to copy, open an interpreter or ChatGPT, paste it, run it, then answer, than just run the code in their head.
Or I can just hold the power button on my android and ask gemini to solve the problem on the screen and sidestep the entire copypasta trap which is less taps and keystrokes than copypasta in the first place
is that not more effort than just solving it?
Give me your email. I'll send you a similar test with a hidden message, take a screenshot and send it to whatever llm, then start reading my message to see which one of you solves it first
I know that I can comfortably follow the logic of a question like this in my head. I also know that, however great I am, I can make mistakes. And the odds of the python interpreter making a similar mistake are effectively 0. So faced with a scenario where I could get a computer to do a calculation, I’m probably going to make it do so.
That said, if the snippet was of this approximate length and complexity, I think I’d spot the discrepancy between what I saw and what was copied. It wouldn’t take many more operators before I wouldn’t be confident in that.
Joke’s on him, I took a screenshot and used a VLM and (of course) it got the correct answer right away.
I solved it in my head… and the first time, got an answer that wasn't even on the list of options. Because I went like "okay 3 is not greater than 3" and then instead of "so that means else" I went with the first branch, because the "not" from the thought process had escaped into my perception of what the code was. Turns out brains are also fuzzy, heh.
A few comments are saying this is illegal and discriminatory to people use screen readers. If anything the opposite may be the case; since it does not affect people using a screen reader they have and advantage that they won't fall into the CTO's trap.
The span with the hidden equals sign includes the aria-hidden="true" which prevents the screen reader from reading it.
<span aria-hidden="true" style="font-size: 0px; opacity: 0; user-select: text;"> = </span>I saw this before and thought it was pretty clever. I'm in IT but solved it first in my head, got it correct, then read about the trick with the hidden equal sign.
Funny thing is if you use snipping tool and screenshot the problem and paste that into ChatGPT, it solves it correctly, lol
Some CTOs will not hire you if you don't use ChatGPT. Some others will not hire you if you do.
What is anybody to do!?
I was viewing it with CSS disabled, so got -11.
Then looking at the "answers" was wondering what he meant by "hidden equals sign", it was only when enabling CSS that I saw what he was referring to.
So if this was in a plain web page, I'd "fail". If it was in a "web app" where I had to have CSS enabled, I'd pass.
I'd have to suggest it is a flawed test.