Having the right prompt is critical to getting the most effective outputs from a generative model. Today, we are excited to introduce Cohere Prompt Tuner, a new tool that can help you optimize prompts right in your Cohere Dashboard.

Prompt Tuner uses a customizable optimization and evaluation loop to refine prompts for generative language use cases. Inspired by Optimization by PROmpting (OPRO), Prompt Tuner iteratively optimizes a prompt based on evaluation results powered by an LLM. The evaluation criteria are customizable, and the results help guide an instruction generation model to generate a new proposed prompt.

Enhancing Model Capabilities

In a typical enterprise LLM workflow, AI engineers spend approximately 20-30% of their time on prompt engineering. Crafting the perfect prompt presents challenges, even for experts, due to the stochastic nature of LLMs. Multiple testing and evaluation rounds are often necessary to refine prompts, ensuring that they operate correctly and are free from biases linked to recent or isolated data examples.

Using Prompt Tuner can lead to a more robust workflow. This tool not only automates the manual aspects of prompt engineering, but it can also unlock model capabilities that can be difficult to achieve through traditional prompt engineering.

In internal evaluations, Prompt Tuner successfully optimized prompts in 94% of use cases. It also proved valuable in facilitating transitions to new models and scaling to new use cases, where the time spent on prompt engineering can further increase to 40%. Optimizing with Prompt Tuner streamlines the development process and boosts overall efficiency.

As seen in the graph below, Prompt Tuner improved an assisted troubleshooting Q&A prompt significantly, improving its evaluation score from 72% to 95% over 10 iterations in a single run.

*Cohere Prompt Tuner improves a prompt’s evaluation score from an initial score of 72% to a final score of 95% after 10 iterations of prompt tuning in a single run.*

Using Cohere Prompt Tuner

Using Prompt Tuner is a systematic and iterative process. Here are the steps to get started:

Set the initial prompt: Create the initial prompt that needs to be optimized and include any variables or documents.
Define success criteria: Establish clear success criteria for the prompt. Choose from several metrics, including accuracy, character count limits, restrictions to JSON output, grounding, and more.
Get the optimized prompt: Watch as the model generates optimized prompts based on the success criteria. The prompt is now ready to be used.

Under the hood, Prompt Tuner is taking the following steps after the success criteria has been defined and the user has started the prompt optimization run.

Generate completions: The target model generates completions for each fully rendered prompt based on the initial prompt.
Evaluate completions: Prompt Tuner submits the generated completions to the evaluation service to get normalized scores across the chosen criteria.
Engage Command R+: Prompt Tuner engages Command R+ as the instruction generation model. Command R+ processes this data to generate an improved prompt that is expected to perform better according to the evaluation criteria.
Iterate: Next, Prompt Tuner extracts the revised prompt template, reinserts the variables, and generates new completions using the target model. This cycle repeats, continuously refining the prompt.
Conclude the optimization loop: The optimization process continues until the prompt achieves a perfect score or reaches the maximum number of allowed iterations. At this point, the system finalizes and delivers the best-performing prompt to the user.

See the steps in action in the diagram below.

Quality Improvements from Prompt Tuning

We completed tests with Prompt Tuner across 41 enterprise use cases, including summarization, customer support, and code generation. For each use case, we defined a set of deterministic and qualitative evaluation criteria that defined success for that task.

Our internal evaluations showed a mean improvement of 11.5% with a median improvement of 7.7%. We observed strong improvements for natural language to SQL and entity extraction use cases specifically. For the SQL prompt, the average prompt score improved from an initial score of 39% to an optimized score of 100%. For entity extraction, the average prompt score improved from an initial score of 73% to an optimized score of 93%.

The entity extraction example can be seen in the table below. The initial prompt is shown on the left and the optimized prompt is on the right. While the original prompt lacked examples, the optimized prompt filled in templates following the requested ISO formats, included marked placeholders for the model to understand and replace with actual values, and also improved the overall readability of the prompt for humans.

Initial Prompt

Optimized Prompt

${document}

You are given a scanned and OCR-ed utility bill. Extract the relevant information and provide it as a JSON object in the following format:

{"billDate": "", "dueDate": "", "billNo": "", "serviceAddress": "", "accountNo": "", "totalAmount": "", "previousBalance": "", "paymentAmount": "", "consumptionCharges": "", "otherCharges": "", "currency": "", "country": ""}

Each extracted field should have "key", "format," and "content" properties. The dates, 'billDate' and 'dueDate,' must be in YYYY-MM-DD format. Ensure all numeric values are converted to strings. If a value is unavailable, use "content": "NotAvail". Provide the "country" in ISO-3166 format and "currency" in ISO-4217 format.

${document}

Extract the following fields from the bill: bill date, due date, bill number, service address, account number, payment details (total amount, previous balance, and payment amount), consumption charges, other charges, currency, and country.

Structure your response as a JSON object with the fields mentioned above, ensuring that dates are in YYYY-MM-DD format, numerical values are extracted accurately, currency follows the ISO 4217 standard, and country adheres to ISO 3166 compliance.

If a value cannot be extracted, use "NotAvail" as the content.

Example response:
{
"billDate": "YYYY-MM-DD",
"dueDate": "YYYY-MM-DD",
"billNo": "BILL_NUMBER",
"serviceAddress": "ADDRESS",
"payments": [
{ "total_amount": "TOTAL_AMOUNT", "previous_balance": "PREVIOUS_BALANCE", "payment_amount": PAYMENT_AMOUNT }
],
"consumption_charges": "CONSUMPTION_CHARGES",
"other_charges": "OTHER_CHARGES",
"currency": "ISO_4217_CURRENCY_CODE",
"country": "ISO_3166_COUNTRY_CODE"
}

Anna Bialas