I ran an interview this week, testing a new process. I told my candidate they could use ChatGPT, Claude, or other tools during the interview.
But instead of Q&A, I ran it as a live dialog on varoius topics specific to their resume and experience, which lined up with what I need for a dev.
Example:
"Design an order processing system."
Then I started throwing curveballs mid-answer:
- It needs offline sync
- Now it's multi-region
- Traffic just went 1000x
- Your vendor is unreliable
- Your queue got breached, business wants a replacement next week - walk me through the plan.
If they knew their stuff this would be a straight forward dialog and the AI would be a tool in the background.
If they didn't, the latency would tell me everything.
In this case the the candidate succeeded, didn't miss a beat, actually told me that AI gave them some suggestions that they didn't agree with and walked me through how they'd work with the LLM to get better responses.
I didn't test their knowledge, but how they think under shifting constraints, which is the actual job.