Opening a web page from Python

MP 137: I just learned about the webbrowser module; maybe it can save you some work as well.

Note: I'll start a new series about debugging on Thursday, April 3. The series will cover a variety of debugging tools and techniques, and will also offer a focused way to practice debugging.

I've often needed to open a web page from within a Python script. Usually, this has come up in projects I'm only running on my own system. In those cases, I've always just written a bit of code that makes a system call to open a specific browser. I recently needed to do this in a cross-platform way, and realized I couldn't count on any one browser to be installed on the user's system. After starting to write a bunch of conditional code, I was quite happy to discover the webbrowser module.

A brittle approach

First, it's helpful to know you can open a web page from within a terminal like this:

$ open -a Safari https://www.mostlypython.com

This will open a new tab in your current Safari window, at the specified URL. You can do this for any browser installed on your system. Here's the equivalent command for opening the same page in Chrome:

$ open -a "Google Chrome" https://www.mostlypython.com

If you know a command that works on your system, you can call it with subprocess:

import subprocess
import shlex

url = "https://www.mostlypython.com"

cmd = f"open -a Safari {url}"
cmd_parts = shlex.split(cmd)

subprocess.run(cmd_parts)

The command we want to run in a terminal is assigned to cmd. Many commands need to be passed to subprocess.run() as a series of arguments. The shlex.split() function splits a command into parts that are appropriate in the context of a terminal. For example, shlex.split() will keep "Google Chrome" as a single argument. If you use the string method split(), arguments like "Google Chrome" will be broken up:

>>> cmd = 'open -a "Google Chrome" https://www.mostlypython.com'
>>> cmd.split()
['open', '-a', '"Google', 'Chrome"', 'https://www.mostlypython.com']
>>> import shlex
>>> shlex.split(cmd)
['open', '-a', 'Google Chrome', 'https://www.mostlypython.com']

This would cause an error, because only part of the "Google Chrome" argument would be passed to the system's open command.

If you run url_opener.py on macOS, you should see this newsletter open in a new Safari tab. But I'm pretty sure most readers aren't on macOS, and there are many macOS users who want URLs opened in a browser other than Safari. The approach shown here worked for me for a long time, but it doesn't work for any of these users.

Try-except and sys.platform ?

When I needed to implement a cross-platform approach, I started writing code that looked something like this:

browsers = ["Safari", "Google Chrome", "Firefox"]
url = "https://www.mostlypython.com"

while True:
    browser = browsers.pop()
    cmd = f"open -a {browser} {url}"
    cmd_parts = shlex.split(cmd)

    try:
        subprocess.run(cmd_parts, check=True)
    except subprocess.CalledProcessError:
        continue
    else:
        break

This code defines a list of browsers that might work on the user's system. It then starts a loop, working with one browser at a time. It defines cmd using the current value of browser, and then tries to open the page using that browser. If it's successful, the else block runs and the code breaks out of the loop. If an exception is raised the loop continues, trying the next browser in the list.

This felt like a lot of work I shouldn't have to do. I could also see how hard it would be to maintain this code. So I decided to look for a built-in solution, or a third-party library.

The webbrowser module

A quick bit of reading led to the built-in webbrowser module. This module does exactly what I was trying to do, in a single line:

import webbrowser

url = "https://www.mostlypython.com"
webbrowser.open(url)

There's no need to inspect the user's system, or try a bunch of browsers. The internal logic of the module takes care of all that for us! It finds the right browser, and runs the correct command to open the URL in a new browser tab.

Peeking inside webbrowser

That was all I needed for the work I was doing, but I was curious to peek inside the webbrowser module and see if it implemented similar logic to what I'd started writing. It turns out it does, in a much more robust way than I could have come up with in a reasonable amount of time.

The current implementation of webbrowser.py is 709 lines long. Here's the implementation of webbrowser.open():

The module defines a variable called _tryorder. This is a list of browsers, in the order they should be tried on a given system. If the order is not defined, it calls register_standard_browsers() to populate that list. It tries to open the URL in each successive browser. It returns True the first time the page opens, and returns False if it's unable to open the page in any of the listed browsers.

Here's the first part of register_standard_browsers():

This code takes into account the user's system, and all the possible browsers that might be installed on that system. This function is about 200 lines long, and shows how much work I would have to do in order to come up with a solution that works for all users!

Conclusions

If you find yourself needing to open a web page from a Python script, save yourself some work and use the webbrowser module.

If you find yourself doing something else that requires a lot of repetitive code, and it seems like other people have probably needed similar code, take a moment to see if there's an existing library that addresses the problem in a well-tested fashion. You might find something in the standard library, or in the third-party ecosystem.

If you do find a library that seems to fit your needs, take a quick look at the internals. You might be validated in your thinking about how to approach the problem, or you might find an entirely different implementation that you hadn't thought of yet.