Large Language Models Have a Hard Time Identifying Missing Context

In my experience using large language models (LLMs), one of my most frequent frustrations is that they seem mostly unable to identify when they are missing some context needed to fully answer a question. In other words, they don’t “know” when to ask clarifying questions (many LLMs seem to be trained to ask follow up questions to an initial prompt, presumably to encourage the user the continue the conversation, but these questions typically seem to broaden the initial prompt scope, rather than attempt to clarify it).

For example, in this chat I had with Gemini, I asked what is the type annotation for self.root in this python Trie class?

class Trie:

    def __init__(self) -> None:

       self.root = {}



def add(self, word: str) -> int:

    node = self.root

    count_prefix_suffix_pairs = 0

    n = len(word)

    for index in range(n):

        character = word[index]

        suffix_character = word[n - index - 1]

        node = node.setdefault((character, suffix_character), {})

        count_prefix_suffix_pairs += node.get("#", 0)

        if index == n - 1:

            node["#"] = node.get("#", 0) + 1

    return count_prefix_suffix_pairs

The LLM returns a confident answer, but doesn’t ask about the environment at all (for example, what version of Python am I using, what static type checker I was using, how is that type checker configured, etc…). Instead, it ends its response by asking if I would like for it to optimize the add method or implement a search function:

Would you like me to help you optimize the add method or explain how to implement a search function for this specific tuple-based Trie?

The recommended type annotation in the initial response

# We define the type alias for clarity
# Key: Union[Tuple[str, str], str] -> (char, char) or "#"
# Value: Union["TrieNode", int] -> Nested dict or the count
TrieNode = Dict[Union[Tuple[str, str], str], Union["TrieNode", int]]

does not work in the environment I was working in. Using Pyright in strict mode with Python 3.11, attempting to use the suggested type annotation will lead to the following errors:

Type of “setdefault” is partially unknown

Type of “setdefault” is “((key: Tuple[str, str] | str, default: Dict[Tuple[str, str] | str, …] | int, /) -> (Dict[Tuple[str, str] | str, …] | int)) | Unknown”

Cannot access attribute “setdefault” for class “int”

Attribute “setdefault” is unknown

Type of “count_prefix_suffix_pairs” is partially unknown

Type of “count_prefix_suffix_pairs” is “Unknown | int”

Operator “+=” not supported for types “int | Unknown” and “TrieNode | int | Unknown”

Operator “+” not supported for types “int” and “TrieNode”

“setitem” method not defined on type “int”

Operator “+” not supported for types “TrieNode | int | Unknown” and “Literal[1]”

Operator “+” not supported for types “TrieNode” and “Literal[1]”

Return type, “Unknown | int”, is partially unknown