Managing two computers

MP 104: Keeping two machines in sync, using Syncthing.

Note: The Django from first principles series will continue next week.

My basic computing setup has changed a lot over the years, but for a while now I've had a pretty simple setup. I invested in a Mac Studio as my main computer a couple years ago. For a while I kept an older MacBook Pro as my backup and travel computer, but last fall I updated that machine to a newer MacBook Air. These are two of the best computers I've ever owned, and I'm pretty sure I'll have both of them for the foreseeable future.

I really like the combination of having a more powerful desktop computer for daily work, and a laptop for working around the house and while traveling. But that brings up the question of how to keep these machines in sync. Are they mirrors of each other? Are there just some directories that are in sync? How do I copy work between the two?

Much of my work is hosted online these days, on places like GitHub and Ghost. But there's still a lot of offline work to keep track of. I recently learned of Syncthing, an open source tool that manages the task of syncing multiple systems. It took a little while to understand Syncthing's workflows and configuration syntax, but I love it now that I have it set up. I'm making better use of my laptop while at home, and working on a wider variety of projects when I travel.

In this post I'll share how I approach syncing between the two systems, and how I set up Syncthing to manage that approach.

Why a desktop?

These days most of my work involves programming, writing about programming, and helping other people learn to program. This means I'm hopping back and forth between macOS, Windows, and Linux on a regular basis. For a while I had a small collection of older laptops, each running a different OS, but that approach wasn't very efficient, and it really wasn't sustainable. I settled on using macOS as my main system, and I run virtual machines on Parallels when I need to use other OSes.

When I was using a laptop as my main system, I had to shut everything else down when I needed to run a virtual machine. Shutting down several projects in order to focus on one task is not very efficient. I found myself avoiding tasks that required a VM, and being slow to resume working on projects I had just shut down in order to run the VM. I wanted a system powerful enough that I could keep all my active projects open at the same time, and run multiple VMs without shutting anything else down.

I finally splurged on an M1 Mac Studio with 64GB of RAM. It's worked exactly as I hoped it would. I've been able to keep all my current writing projects open, and each of those projects usually involves a bunch of programming files as well. I usually have an open source project or two open, and I have a general-purpose workspace set up all the time as well. With all these projects open, I can still run several virtual machines, and keep them running as long as I need to do testing on other OSes, without affecting the performance of other workspaces at all. I have eight workspaces set up, and most of them are usually full.

As much as I love the Studio, I can't bring it downstairs when I need to hang out with the puppy, and I can't bring it with me when I travel. So I end up using the laptop a fair bit as well.

Before Syncthing

Before using Syncthing, I had a folder on my laptop called local_mac_work/. I'd use AirDrop to copy the projects I wanted to work on from the Studio to the laptop, and then copy those back to the Studio when I returned to my desk. This sort of worked, but it meant I avoided doing too much on my laptop because I didn't want to lose track of what files I had changed. I also couldn't work on any of the projects I hadn't copied over to the laptop.

I wanted a better approach, and when Jeff Triplett shared his experience with Syncthing, I made a note to set it up when I had a chance. My use case is simpler than Jeff's; he was syncing several computers, and I just have two to keep track of.

Using Syncthing

The first task in setting up Syncthing was to decide exactly what I wanted to sync. The Studio hard drive is larger than the laptop's, and I don't want to just sync the entire system. I settled on syncing the Desktop/ directory, my projects/ directory, and a few other specific directories.

I use my Desktop/ folder as a scratch area, and as a place to hold a few files and folders that I access every day. Most of my active projects are stored in projects/; if this folder is synced, I'm much more likely to jump into any project that comes to mind, especially while traveling. I'd much rather do a bit of work on a project while traveling, than make a note to do that work when I get back home. I also have a few more directories I want to sync, related to things like writing, photos, and outdoor stuff.

Back up everything

As always when setting up a tool that's going to manage your system, you really should start by making sure all your backups are current. I rotate between two external drives, so I made sure both drives had a current backup. I usually keep one backup drive connected at all times, but I disconnected both during this setup work. I didn't want to accidentally have Syncthing erase this drive, or try to sync the backup drive to my laptop.

Also, if you're syncing a large amount of data to a mostly-empty secondary system, I encourage you to do that initial copying work from a backup before using Syncthing. Syncthing is great for syncing systems that are mostly up to date with each other, but it can be slow if it has to copy a large amount of data.

Initial setup

I used the Getting Started guide from the Syncthing documentation. I installed Syncthing using Homebrew, which was straightforward. You need to install Syncthing on both systems, but I started on just the desktop.

Once Syncthing is installed, you have to start it. I ran it as a service:

$ brew install syncthing
...
==> Caveats
To start syncthing now and restart at login:
  brew services start syncthing
Or, if you don't need a background service you can just run:
  /opt/.../bin/syncthing -no-browser -no-restart
$ brew services start syncthing
==> Successfully started `syncthing`

This makes the admin GUI available at http://localhost:8384/. Here's what you should see after clicking through the popup about anonymous usage reporting:

Syncthing initially sets up a shared folder named Default Folder. This isn't tied to any folder on your system; it's just meant to be a shared folder that you can modify if you want to start from a template. I chose to remove this folder, and start from scratch. To remove the default folder, click on it and then click Edit, and you should see an option to remove it.

Shared folders

Syncthing works on the basis of "shared folders". You define a folder that's going to be shared between devices, and add it to Syncthing. Both systems need to agree with the configuration about exactly what should be shared for any syncing to happen. Within the shared folder, you can designate which files and folders should be synced, and which should be ignored.

Right away, this brings up a couple different approaches to using Syncthing. You can either share one main folder, and use includes and excludes to specify exactly what's synced. Or, you can share a bunch of specific folders. To keep things simple, I'm using the first approach. I'm sharing my home folder, specifying exactly which items to sync, and then ignoring everything else.

To do this, I clicked Add Folder, and set Folder Label to "Home". Note that this doesn't share my actual home folder. This just sets up a Syncthing shared folder called Home. What matters is the value for Folder Path, which I set to ~:

Start with Send Only and Receive Only folders

When you add a folder, under the Advanced tab, you can choose how this folder will be synced. It can be set to Send & Receive (the default), Send Only, or Receive Only. My folders are set to Send & Receive, because I want to be able to work on either system, and then sync those changes back to the other.

However, when doing the initial configuration work, I set the shared Home folder on my Studio to Send Only, and the shared folder on my laptop to Receive Only. That way if I messed anything up I wouldn't accidentally write over the Studio's home drive. If I messed up the laptop in this initial sync, I was prepared to rebuild that system.

Global ignores

There are some directories and files that should never be synced. For example I don't want to sync virtual environments, or the ubiquitous .DS_Store.

In each shared folder, Syncthing looks for a file called .stignore. That file can include other files, so I started by creating a file called .stignore-global. This creates a nice separation between things that should always be ignored, and configuration for a specific folder that's being shared. Also, if you have multiple shared folders, you can include .stignore-global in each directory's .stignore file.

Here's the contents of ~/.stignore-global:

.venv/
*_env/

dist/
build/
*.egg-info/

(?d)__pycache__/

(?d)*.pyc
(?d)*.pyi

(?d).DS_Store
Thumbs.db

Here I'm ignoring all virtual environments, auto-generated distribution and build files, Python's cache directories and files, and some macOS cruft files. If you're on Windows or Linux you'll probably have a slightly different global ignore file, with some OS-specific cruft you want to ignore.

The (?d) prefix in front of some entries tells Syncthing that it's okay to remove these files and directories if they're found inside a folder that should be synced. It took me a while to figure this out; I had some folders that weren't being deleted on the laptop, after deleting them on the Studio. It turns out Syncthing didn't want to delete the directories on the laptop, because .stignore-global says to ignore them. The (?d) directive says it's okay to delete them within directories that are being synced. If you decide to use Syncthing, definitely take a look at the documentation page about Ignoring Files. I won't be surprised if I end up having to add this prefix to more of these entries at some point.

Notice that .git/ is not listed in .stignore-global. I definitely want to sync my Git directories. Some people choose not to sync their Git directories, but I find it extremely useful to be able to drop into exactly the same state of a project on each system. This is almost certainly easier on a two-system setup than on a setup with more than two computers.

The .stignore file

Syncthing looks for a .stignore file in the root of the path specified in Folder Path, for each shared folder. Since I'm sharing my home drive, this is especially important to get right; I don't want to write all my Studio's dotfiles to my laptop. The two machines have different configurations, and I'm certain it would cause problems to write the Studio's entire home drive to the laptop.

Here's what my ~/.stignore file looks like:

#include .stignore-global

// Only include these specific directories.
!/Desktop
!/conferences
!/my_photos
!/outdoors
!/pcc_3e
!/pcc_3e_staging
!/piano
!/projects

// Ignore everything else.
*

The first line includes the .stignore-global file discussed earlier. All paths in this file are relative to the shared folder path. In this case it's the home drive that's shared, so this is effectively including ~/.stignore-global. If you're sharing something other than your home drive, make sure you get these paths right!

Something that confused me when setting up Syncthing was the use of !, which usually means not, to indicate directories that should be included. I think this comes from the fact that this is an ignore file, and we're telling it not to ignore these directories. Syncthing doesn't have an include file, so this is how you specify an allow list with Syncthing. Here I'm telling Syncthing I want to sync my Desktop/ directory, my notes about conferences, my photos directory, some Python Crash Course directories, and a few others including projects/.

Notice the leading slash for each directory that's included in the sync. The sync would probably work without it, because all paths are relative to the shared folder path. But consider the folder !/projects. The leading slash here says to include the folder projects at the root of the shared folder. If you left out the leading slash, any folder with the name projects in an otherwise ignored folder, such as ~/house_work/projects, would end up getting synced. From the documentation:

A pattern beginning with / matches in the root of the synced folder only./foo matches foo but not subdir/foo.

I want to ignore everything else in my home directory. The single * at the end of the file tells Syncthing to ignore everything that wasn't explicitly included earlier in the file.

You can edit this file in any text editor you like, or you can edit it directly in the GUI, under the Ignore Patterns tab:

With the ignore file complete, I moved over to the laptop.

Setting up the second system

Now I need to install Syncthing on the laptop, and configure it as well. I used Homebrew to install Syncthing on my MacBook Air, and started it as a service just as I did on the Studio. I also removed the default shared folder. You don't need to set up a shared folder on the second machine; we'll initiate that from the first machine.

I'm setting these two systems up for bi-directional sharing, so the laptop needs the same .stignore-global and .stignore files that the Studio has. I used Airdrop to copy the files between the two systems. Don't skip this step, or when you set both systems to Send & Receive the laptop will try to share too much with the desktop.

Adding remote devices

Syncthing generates an ID for each device. In order for the sync to work, each device needs to be set to trust the other device's ID. This means device IDs don't need to be kept secret. From the Getting Started guide:

Since the configuration must be mutual for a connection to happen, device IDs don’t need to be kept secret. They are essentially part of the public key.

You can see a device's full ID by clicking on the first part of the ID under This Device. You can copy the ID and then share it to your other device. However, if you wait a few minutes the devices will each notice each other's IDs and you won't have to deal with copying the IDs.

Now it's time to actually connect the two devices. On the studio, I clicked Add Remote Device:

You can either click on the ID of a nearby device, or enter a Device ID in directly. You can also give each remote device a friendly name.

My laptop noticed that the Studio added it as a remote, and I can complete the connection by clicking the Add Device button on the laptop:

I was clicking around a bunch while writing this, and Syncthing started scanning my system before I was finishing writing the ignore files, so I paused the scan. I restarted the initial scan by clicking Resume:

Syncthing isn't actually syncing the systems yet. It's just scanning each system, so it knows what data it will need to sync.

Sharing and syncing

The final step to start the initial sync is to tell the first machine to share a folder with the second machine. Go back to the shared folder, click Edit, and click the Sharing tab:

I checked Eric's MacBook Air, and clicked Save. That caused a popup on the laptop:

Clicking Add in this dialog opens a popup where you can edit the shared folder on the second system. The only thing I changed on the laptop was in the Advanced tab, where I set the folder to Receive Only. Clicking Save starts the actual sync.

Note: I noticed later that syncing was not working. When Syncthing shared the desktop's folder with the laptop, it suggested a different path on the laptop: ~/Home. This made a new folder on my laptop, which was a subdirectory of my home folder called Home, with a capital H. All my desktop's files were syncing to that Home directory, which I never use. I rebuilt everything, making sure that the actual path being shared on each system was correct.

When the sync is complete, you should see a message that both systems are "Up to Date":

The initial sync is complete; you should verify this by checking that some files that were only on the first machine are now on the second machine as well. You should also verify that ignored files and directories were not synced.

Ignoring subdirectories

My initial sync worked well. All the files I cared about on my Studio were synced to my laptop. However, there were two subdirectories I wanted to exclude from subsequent syncing. The first was Destkop/screenshots. I keep a folder for screenshots on each system's Desktop/ directory, but there's no reason to keep these folders synced. Also, I sometimes have temporary development projects like projects/dsd-dev-project_305af/ in my projects/ folder.

To exclude these two subdirectories, I added them near the top of .stignore:

#include .stignore-global

// Ignore specific subdirectories.
/Desktop/screenshots
/projects/dsd-dev-project_*

// Only include these specific directories.
!/Desktop
...

It took me a while to figure out that these lines need to go after the line including the global ignore file, but before the list of directories to include. I originally placed these two lines after the list of directories to include. However, Syncthing prioritizes entries that come first in the ignore file. So if the line including /Desktop comes before the line excluding /Desktop/screenshots, then the screenshots directory will be included. I finally realized I needed to exclude /Desktop/screenshots before including /Desktop.

Send & Receive

Now that the initial sync from the Studio to the laptop worked, I was ready to set both systems to Send & Receive. I wasn't too worried about this step at this point. The files that were being synced are now on both laptops, so not much should happen here. I also knew I had fully up-to-date backups when I took this step for the first time.

Click Edit on the shared folder on each system, click the Advanced tab, and set Folder Type to Send & Receive. Click Save, and it should start a new scan, and then show everything as Up to Date again.

Ongoing syncs

If you're syncing two systems that are always on, they'll be synced as soon as Syncthing's watcher notices changes to files and directories. (See the Watch for Changes area in the Advanced tab of the shared folder.) Also, if both systems are on, Syncthing will run a full scan at regular intervals, according to the Full Rescan Interval setting.

My systems aren't usually awake at the same time. When I'm working on my desktop, my laptop is usually closed. When I'm working on my laptop, my desktop is usually asleep. To sync the two systems I bring my laptop to my desk, make sure they're both awake, and click Rescan All on one of the systems. This is a perfectly reasonable workflow for me, because I can always bring my two systems near each other when I need to sync them. There are lots of ways to use Syncthing though, so if this workflow doesn't make sense for you I encourage you to read about the different ways syncing can be configured.

Note that you'll have to keep the .stignore and .stignore-global files in sync manually. There are a number of reasons for this, which you can read extensive discussions about in Syncthing's forum and issues. Briefly, it's prone to cause cascading sync failures across multiple nodes in a system. I'm quite happy to keep these files up to date manually, and be confident that I know exactly what's being synced and what's being ignored.

When you run your first few syncs, keep an eye on all the information Syncthing presents in the GUI. If you see any files listed under Out of Sync or Failed Items, click on them and read the messages Syncthing provides. These messages helped me refine my ignore files until they were making Syncthing behave exactly as I wanted to. I've also found the Recent Changes helpful to look at on both systems at times.

I'd caution people to be wary of advice from GPT or other LLM assistants when working with Syncthing. When I tried using it to troubleshoot my ignore files, it gave me advice that would have had pretty frustrating impacts, like syncing all my virtual environments. That would have involved syncing about 100k extra files, and I would have had to rebuild all those virtual environments on the machine they got copied to.

Conclusions

I've been using Syncthing for just over a month, and it's been fantastic. I'm using my laptop more because it has everything I need, and I don't have to think about how I'll sync my changes at the end of a work session. I feel less tied to my desk at home, and I work on a greater variety of projects when I travel. I'm also packing up my desktop for a cross-country move this summer, and I already know that re-syncing when I set it back up will be a breeze.

Syncthing is fantastic software, and I'm rather surprised I hadn't heard about it earlier. Thank you to Jeff for sharing your setup, and thank you to everyone who works on Syncthing.

With something as important as syncing, you should absolutely look through the documentation before following any of the suggestions in this post. I'm still quite new to using Syncthing, and I'm sure I don't have the most optimal setup yet. That said, I hope reading about my experience makes you more comfortable getting set up with Syncthing on your own system, if you choose to use it.