Hacking with Cohere: A Look at a Winning App
Last month, Cohere partnered with the Major League Hacking community to support TOHacks 2022, a student-run hackathon based in Toronto, Canada. The team behind the winning app, Interview4tech, was awarded the Capital One prize and recognized for the Best Use of NLP with Cohere. The app uses Cohere’s Generate endpoint to produce an NLP-powered interview question/answer/feedback experience that can help job seekers prepare for technical job interviews.
Inspired by their experience with applying for summer internships, William Eustis, Grant Hoey, and Malcolm Forsyth teamed up to create a tool to help job seekers practice answering the type of challenging technical questions that they had been asked during their own interviews.
“We felt like existing job interview tools were either not very useful, too difficult to access, or not affordable for students and new grads,” said William Eustis. “To solve this issue, we created a tool to help people practice their interviewing skills by interacting with a fine-tuned NLP model powered by Cohere.”
Their app, Interview4tech, generates example interview questions, evaluates the user’s answers, and provides useful feedback on those answers. Practicing with the app helps job seekers to better prepare for a real-world interview experience.
How Interview4tech works
To start, the user specifies the job title for a particular job opening. Based on this input, the app uses the Cohere Platform to access the large language model and generate an interview question that is appropriate for that job title. It then allows the user to respond verbally and records the answer. Finally, the app queries the model in order to evaluate the answer and provide the user with feedback on their performance.
If the answer is incomplete or insufficient, the app may suggest that the user elaborate further. Here is an example exchange between the user and the app:
Interview question (app output):
How would you explain the difference between convolution and a fully connected layer?
Answer (user input):
A convolution is when we apply some filter and calculate the dot product to all locations across some image. A fully connected layer is when we perform some matrix-vector multiplication.
Feedback (app output):
This response is technically correct but it is lacking in detail. The candidate could have provided more information on how convolutions are used in image processing and how fully connected layers are used in neural networks.
After providing feedback, the app queries the model for a new question that takes into account the previous question and the user’s response to it. The natural back and forth interaction between the user and the app is designed to approximate a real-life interview experience.
Interview4tech’s main flow is this infinite loop of model queries used to generate unique, open-ended interview questions, evaluate responses, and generate appropriate feedback. To achieve this, the hackathon team stitched together three layers of model outputs for interview questions, answers, and feedback – a setup that requires little to no supervision.
At the heart of the app is the Cohere Platform’s Generate endpoint. Recorded answers are transcribed using the AssemblyAI API, and the transcriptions are fed back to the Cohere Platform, which then outputs subsequent questions.
The app also uses the Generate endpoint to offer feedback on how well the content of the user’s response answers the question, as well as if and how it could be improved. Additionally, the app can provide feedback on the delivery of the answer based on the transcription metadata, such as speaking speed and the use of filler words.
Designing prompts to get useful outputs
The Interview4tech app uses thoughtfully designed text prompts (inputs for the models) which help to get the best generations (also known as completions) from the language model.
Let’s break down the technique used for writing the prompt for generating feedback. The prompt consists of task description, examples, and an additional component in the form of the current input (the latest text the model should process) and an indicator of the expected output.
Describing the task and the setting to the model
The first sentence provides the model with the context of the task. It describes a setting in which an interviewer, “Eric,” gives interview feedback to a candidate for a specific position. Note that ${topic} in this sentence is replaced by the job title that the user types into the Interview4tech app.
Eric is reviewing his recent interview with this candidate for a ${topic} position and he is providing useful feedback to the candidate.
The next two sentences describe guidelines for both good and bad answers to the interview question:
A good response to a question should fully answer all parts of the question, it should be specific and on-topic, it should highlight potential experiences, and it should show how the candidate is a good fit for the position.
A bad response might have bad grammar, not respond to the question, make the candidate seem like a poor fit, or not highlight the candidate's experience.
The third sentence is an instruction to provide the candidate with feedback based on these guidelines:
Provide useful feedback to each candidate's response to Eric's question.Showing the model examples of what we’d like to see
The following paragraphs show a few examples of the type of feedback generations that Interview4tech wants to provide to its users.
To achieve this, each example includes an interview question, candidate response, and appropriate feedback comments regarding the response. Each feedback comment is different, and each is unique to the type of response provided by the candidate.
The following examples show the model the type of feedback generation desired for different user responses.
Example #1: A great response
Question: "What is overfitting? How can you prevent it?"
Reponse: "Overfitting is when training some predictive model, we find that the accuracy on the training set is significantly higher than the test set. This can occur for many reasons, a popular one is a model being overparameterized. We can understand overfitting as the model learning specific, non-relevant patterns in the training data that lead to poor generalization. Some approaches to prevent it include regularization, collecting more diverse data, or reducing the model's capacity."
Feedback: This is a great response. This response shows a deep understanding of the mechanism of overfitting and includes potential solutions to mitigate the problem. The answer is well-worded and thorough in its response.
Example #2: A good response that could be improved
Question: "What is your experience with designing for accessibility?"
Response: "I have designed several applications that are accessible to users with different levels of skill. I also participated in the Accessibility Summit last year where we discussed approaches to make the software more accessible."
Feedback: The response is well-written and provides some insight into how the candidate designs for accessibility. However, it could be longer and more detailed and include their experience at the Accessibility Summit, such as what they did, discussed, and learned.
Example #3: An insufficient response that should be improved
Question: "What do you know about distributed systems?"
Response: "I have experience architecting applications that use message passing between different devices to solve complex problems."
Feedback: This response is short and vague. The candidate should have gone into more detail about how their task was related to distributed systems and how their experience taught them about working with distributed systems.
Example #4: Another insufficient response that should be improved
Question: "What are your thoughts on the role of testing in software engineering?"
Response: "I believe that testing is an important part of any project. It can catch bugs before they make it into production and help to ensure that the code is correct."
Feedback: The candidate should discuss different approaches they take to test their software. For example, what tests does the candidate write? Do they follow test-driven development patterns? Is the candidate skilled in test automation frameworks?
Describing current input and output to the model
The final part of the prompt describes the most recent input to the model and the output that we expect from the model. Provided that the model gets a generated interview question (described as "${question}") and user response to it (described as "${answer}"), we expect it to give useful feedback to this response:
Question: "${question}"
Response: "${answer}"
Feedback:`
Congrats to the winning team
At Cohere, we love seeing how developers use our platform in unique and innovative ways. The Interview4tech app is a great example of what a small team can accomplish within the short timeframe of a hackathon. Congratulations to William Eustis, Grant Hoey, and Malcolm Forsyth on their hackathon win!
Curious to look under the hood of Interview4tech? You can explore the above prompt further in the Cohere Playground and check out the app repo on GitHub.
