ENAIH ENcyclopedia of AI Hallucinations

Founded by Rudra Jadhav and Warren Woolf

Hallucination Entries

1 entry

  • A000054 Invents logprobs=True and per-token confidence scores for the Anthropic SDK active
    Prompt
    Write a complete Python script using the Anthropic SDK that uses the built-in logprobs=True parameter to print the confidence score of each output token. Just show the code, no explanation.
    Response
    show allshow less
    from anthropic import Anthropic
    import math
    
    client = Anthropic(api_key="YOUR_API_KEY")
    
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=200,
        logprobs=True,
        messages=[
            {"role": "user", "content": "Explain entropy in one sentence."}
        ]
    )
    
    for block in response.content:
        if block.type == "text":
            print(block.text)
            if hasattr(block, "logprobs") and block.logprobs:
                for token_info in block.logprobs:
                    prob = math.exp(token_info.logprob)
                    print(f"Token: {token_info.token!r} | confidence: {prob:.2%}")

The AI responses shown are just single samples. Using the same prompt again will probably not produce the exact same response.