LLM Engineering

If a single answer is right 60% of the time and samples were independent, a majority vote over 5 samples would be right 68.3% of the time and over 15 samples 78.7%. That is the idea behind self-consistency. In practice samples share their mistakes, so the real gain is smaller — but the direction holds.

• Results are sensitive to small formatting changes — spacing, separators, option order — sometimes by many points (Sclar et al. 2024).
• A prompt tuned on one model version often does not transfer to the next.
• Prompt tweaks evaluated on a handful of examples are a garden of forking paths; test on a held-out set.

from math import comb
def majority(p, n):                               # independent samples, binary right/wrong
    return sum(comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n // 2 + 1, n + 1))
votes = {n: round(majority(0.6, n), 3) for n in (1, 5, 15)}

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

from openai import OpenAI
client = OpenAI()

# Chain-of-thought with structured output
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a math tutor. "
         "Always work through problems step by step. "
         "Return JSON with 'steps' (array) and 'answer' (number)."},
        {"role": "user", "content": "If a train travels 120km in "
         "1.5 hours, what is its speed in m/s?"}
    ],
    response_format={"type": "json_object"},
    temperature=0.1
)Copy