Parsing large(ish) data files

MP 173: It's always an interesting challenge.

As I've been getting back into chess, I've enjoyed exploring the ever-growing public database of games people have played. But some of that exploration has been slowed by not having an established approach to parsing large numbers of games.

There are many tools for exploring game collections, but I've just started to become familiar with these tools. I'm guessing people have made tools for exploring large collections efficiently, but I haven't found them yet. I'll do a more thorough search for existing tools when I have a bit more time, but for now it's kind of fun to come up with my own ways to handle these larger files.

In this post, I'll share the challenge I've come across with a naive approach to parsing large game files, and share a more efficient approach I've been curious to explore.

Playing the English

Earlier in my chess-playing life, I chose a couple openings and stuck to those for every game I played. I thought that if I stuck to just a few openings, I'd learn all the tricks and traps in those lines and improve faster. I wasn't trying to play tricks against opponents; I was trying to build a deep understanding of a few openings.

That worked for a while, but it was starting to make my games feel somewhat stifling. Now that I'm playing the same people regularly at a local coffee shop, they're getting used to me playing these same lines over and over again. What's fun for me against an entire internet of chess players is not as fun against people I'm becoming friends with.

With this realization, I've been trying other openings, and it's been way more enjoyable than I ever thought it would be. Instead of feeling lost in these new openings, I feel like I'm seeing the board with fresh eyes. And when I go back to the openings I have more experience with, I'm understanding the ideas in those openings better than I did before because I have something to compare them against.

My new favorite opening as White is the English, which starts off with 1. c4:

I'm really enjoying this opening because the style of play it leads to is significantly different than the King's Gambit, which was my favorite White opening for a long time.

I'd like to find recent higher-level English games, so let's try to find them in some game collections.

Building some datasets

The Lichess database files are large; compressed, they're about 30GB each. However, they support partial downloads. I started a download of the games from July 2026, and generated three files of roughly 1k games, 10k games, and 100k games:

$ ls -alh data
2.2M Aug  5 21:06 lichess_2026-07-1k_games.pgn
22M Aug  5 21:08 lichess_2026-07-10k_games.pgn
222M Aug  5 21:09 lichess_2026-07-100k_games.pgn

This is a small subset of the games that are available; there are almost 100 million games in the full July 2026 dataset! We have to figure out how to work with these three files efficiently if we want to have any hope of exploring the entire dataset.

Using the chess package

If you try to work with chess games in Python, you'll quickly find the python-chess project. It's built for working with chess games, so let's use it to count how many English games there are in the file containing 1,000 games:

"""Parse large-ish pgn files."""

from pathlib import Path
import sys
import chess.pgn

path = Path(sys.argv[1])

games_examined = 0
english_count = 0

with open(path) as pgn:
    while game := chess.pgn.read_game(pgn):
        opening = game.headers["Opening"]
        if opening.startswith("English"):
            english_count += 1

        games_examined += 1
        if games_examined % 1_000 == 0:
            print(f"\nGames examined: {games_examined}")
            print(f"English games found: {english_count}")

print(f"\nGames examined: {games_examined}")
print(f"English games found: {english_count}")

This short program lets you pass in a path to a data file. It opens that file using the chess library, and loops over every game in the file. It then examines the opening, which is accessible through game.headers["Opening"]. If the opening starts with English, we increment english_count.

I know I'm going to examine larger files, so I added a block to track how many games have been examined, and report the progress every 1,000 games. At the end, we report what was found.

Here's the output:

$ time uv run main.py data/lichess_2026-07-1k_games.pgn 
Games examined: 999
English games found: 34
0.932 total

This seems correct; Lichess' opening explorer reports that about 3% of all games in the database are English games. These numbers have some nuance; are you only counting games that start with 1. c4, or are you also counting games that start with something else but transpose back to English positions?

The program found 34 English games out of 999 total games, and finished in just under 1 second. Let's see how it does against 10k games:

$ time uv run main.py data/lichess_2026-07-10k_games.pgn
...
Games examined: 9992
English games found: 355
8.129 total

It still seems to be working; it found 355 English games out of 9,992 total games, in just over 8 seconds.

And now let's try to parse 100,000 games:

$ time uv run main.py data/lichess_2026-07-100k_games.pgn
...
Games examined: 99929
English games found: 3674
1:22.03 total

It's still finding about 3% of the total games, but it took about 82 seconds to parse the 100k games. That's really starting to slow down; I wouldn't want to run this program over 100 million games!

Let's try a different approach.

Chess-agnostic parsing

These PGN files are just giant text files with a bunch of chess-specific information in them. Here's what an English game looks like in any of these PGN files:

[Event "Rated Blitz game"]
...
[Opening "English Opening"]
...

1. c4 { [%clk 0:05:00] } 1... a6 { [%clk 0:05:00] } 2. ...

The headers have an entry for the opening, and the game is a single line of moves. The clk entries are timestamps, showing how long players took for each move.

Let's just parse this text, looking for Opening "English. The following program uses a regular expression to find those instances:

"""Parse large-ish pgn files."""

from pathlib import Path
import re
import sys

path = Path(sys.argv[1])
contents = path.read_text()

re_english = r'''(Opening "English)'''
m = re.findall(re_english, contents)

print(f"English games found: {len(m)}")

We read in the contents of the entire PGN file as a string, and then call re.findall() with a regular expression that looks for Opening "English. We report the number of matches that were found.

Let's see how it does on the file with 1,000 games:

$ time uv run main_regex.py data/lichess_2026-07-1k_games.pgn
English games found: 34
0.058 total

It found the same number of games as the previous approach, in about 0.058 seconds. That's about 16 times faster.

Here's 10k games:

$ time uv run main_regex.py data/lichess_2026-07-10k_games.pgn
English games found: 355
0.080 total

It found the 355 English games in 0.080 seconds, 101 times faster than the other approach.

And here's 100k games:

$ time uv run main_regex.py data/lichess_2026-07-100k_games.pgn
English games found: 3674
0.304 total

It found the 3,674 English games in 0.3 seconds, 273 times faster than the original approach.

This isn't particularly surprising. We're not building any Python objects, we're just doing text parsing. It's interesting to note that the speedup is getting more significant as we process greater numbers of games.

Conclusions

Domain-specific libraries like python-chess are fantastic tools. But sometimes we need to do some more general work before using these kinds of tools.

I'm going to follow up on this in a couple ways. First, I want to do more than just count how many English games have been played recently. I want to look at the most interesting recent English games. I think I'll do this by filtering games as much as I can with general programming techniques, and then use a domain-specific tool like python-chess when I need to. The goal will be to write a much smaller PGN file that I can load into a Lichess study, and examine those games to help my own play.