Gemini Contractors Might Be Rating AI Prompts Outside Their Expertise

Google is reportedly asking contractors working on evaluating Gemini’s responses to rate prompts outside of their domain of expertise. As per the report, the Mountain View-based tech giant has removed the option to skip prompts, which was exercised by these contractors if they felt they did not have enough knowledge about a subject to rate the response. With artificial intelligence (AI) hallucinations as a major concern for chatbots, this reported development could lead to a drop in the quality of Gemini’s responses when it comes to highly technical topics.

Google Reportedly Not Letting Contractors Skip Gemini Prompts

According to a TechCrunch report, Google has sent a new internal guideline to contractors working on Gemini. Claiming to have seen the memo sent by the tech giant, the publication claims that these contractors are now being asked to answer queries even when they might not possess the knowledge to correctly assess the answers.

Google reportedly outsources the evaluation of Gemini’s responses to GlobalLogic, a firm owned by Hitachi. The contractors working on Gemini are said to be tasked with reading technical prompts and rating the AI’s responses based on multiple factors such as truthfulness and accuracy. These individuals evaluating the chatbot hold expertise in specific disciplines such as coding, mathematics, medicine, and more.

So far, the contractors could reportedly skip certain prompts if it was outside of their domain. This ensured that only those qualified to understand and evaluate technical responses generated by Gemini were doing so. This is a standard post-training practice for foundational models and allows AI firms to ground their responses and reduce the instances of hallucination.

However, this changed when GlobalLogic reportedly announced the new guidelines last week that contractors were no longer allowed to skip prompts unless the response was “completely missing information” or it contained harmful content that requires special consent forms to evaluate.

As per the report, the new guideline states that contractors should not “skip prompts that require specialised domain knowledge” and instead, they should rate the parts of the prompt they understand. They were reportedly also asked to include a note mentioning that they do not have the domain knowledge.

One contractor stated in internal communication, “I thought the point of skipping was to increase accuracy by giving it to someone better,” the publication claimed.