Sets vs tuples for membership tests

MP 178: Which is better?

One of the interesting things about working with agents is getting to see how the generated code differs from the code you might have written yourself. When I generate code with an agent, it tends to fall into one of three buckets:

• It's almost identical to what I would have written.
• It's different than what I would have written, but functionally equivalent and the differences aren't that interesting.
• It takes a noticeably different approach than I would have, and I need to evaluate whether to accept the code or use the approach I would have taken.

It's that last bucket that's been keeping programming interesting for me lately, even when using agents.

Valid operations

I've been using my own agent harness, because I want to know exactly what its boundaries are and how they're implemented. The core logic in the tool involves validating the operations a model is requesting to carry out, and the target files and directories it's requesting to work with.

At one point in a recent development session, this is the code I expected to see:

if operation not in ("read", "modify", "write"):
    raise ValueError(f"Illegal operation: {operation}")

Instead, the model returned this code:

if operation not in {"read", "modify", "write"}:
    raise ValueError(f"Illegal operation: {operation}")

It's the same validation logic, but the model is checking for membership in a set rather than in a tuple.

I asked the model why it chose a set for this validation logic instead of a tuple. It replied that a set is a collection of unique values, which is exactly what we're testing against here. It also said that testing for set membership is O(1), while testing for membership in a tuple is O(n). It acknowledged that there's probably not a lot of difference in performance for such small sets and tuples.

Comparing sets and tuples

When I first learned Python, I fell in love with lists and used them for everything, including simple membership tests like this. I then learned about tuples and started using them for these situations. I learned about sets a little later, and never really considered them for this use case. I feel like I've seen tuples used for small membership tests, particularly in validation logic, in most projects I've looked at.

Let's do some benchmarking. We'll just use the timeit module for now:

$ uv run python -m timeit "operation='read'; valid = operation in ('read', 'modify', 'write')"
20000000 loops, best of 5: 10.4 nsec per loop

Python ran this snippet in a loop 20 million times, and found that it took 10.4 nanoseconds to run the validation test when using a tuple.

Let's try it with a set:

$ uv run python -m timeit "operation='read'; valid = operation in {'read', 'modify', 'write'}"
20000000 loops, best of 5: 11.2 nsec per loop

The same snippet took 11.2 nanoseconds when using a set. My approach won!

Python versions

This first benchmark was run using Python 3.14.7. In the quest to build a performant and useful language, Python evolves with a never-ending series of tradeoffs. Sometimes what's faster in one version might not be faster in the next version. Let's see how this result holds across several recent Python versions:

The tuple approach is marginally faster in Python 3.15, 3.14, and 3.12. The set approach was slightly faster in 3.13 and 3.11.

Conclusions

For most of my work, it really won't matter if I'm choosing to use a tuple or a set for membership tests. I'm going to keep using tuples, because it's what I'm used to and it works. But it brings up some interesting points.

First, AI-generated code isn't always better than what you'd write. Most models can come up with a reason why they chose a certain approach. That reasoning is influenced by many different factors, some of which are relevant to the current use case, and some of which are not.

Second, profiling matters. If you're working with a specific version of Python, make sure you do your profiling with that version. Don't assume the same performance results on one version of Python will apply to the next version, particularly as new features like free-threading become more established. You can keep things simple by always profiling in your current environment before making optimization decisions.