Btrfs v1 to v2 and Docker's Hidden Mounts

I recently spent more time than I would like to admit trying to convert a Btrfs filesystem from free-space cache v1 to v2. The conversion appeared to fail silently, and the usual tools for diagnosing a busy device gave no useful output whatsoever.

Background

The filesystem in question was a two-device Btrfs RAID1 array:

• /dev/sda — 1.82 TiB
• /dev/sdc — 2.73 TiB

Inspecting the superblock with btrfs inspect-internal dump-super showed:

1
2

incompat_flags           0x161
                         ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA )

No FREE_SPACE_TREE in the flags. The filesystem was using the legacy free-space cache v1, which gets stored as regular files in each subvolume rather than as a dedicated tree structure. Converting to v2 is generally recommended as it is more robust and survives unclean shutdowns better.

First Attempt

Mounting with the conversion flags seemed straightforward enough:

1

sudo mount -o space_cache=v2,clear_cache /dev/sdc /mnt/btrfs

The filesystem mounted without errors. But checking the superblock again showed incompat_flags 0x161 — unchanged. No FREE_SPACE_TREE. Trying again without clear_cache made no difference.

btrfs-progs 6.17.1 ships a more direct tool for this:

1

btrfs rescue clear-space-cache v1|v2 <device>

So I unmounted /mnt/btrfs and ran:

1

sudo btrfs rescue clear-space-cache v1 /dev/sdc

The result:

1
2

ERROR: cannot open device '/dev/sda': Device or resource busy
ERROR: cannot open file system

The Puzzling Part

This is where it gets frustrating. The error says the device is busy, but nothing confirmed that:

• fuser /dev/sda /dev/sdc — nothing
• lsof /dev/sda /dev/sdc — nothing
• findmnt — /mnt/btrfs was not listed
• No device-mapper devices existed
• Both block devices were writable

I had also paused the Docker containers running on the host. Or so I thought.

Finding the Culprit

The key insight is that fuser, lsof and findmnt all operate within the context of the host's mount namespace. Docker containers run in their own separate mount namespaces. A filesystem mounted inside a container's namespace is invisible to these tools from the host's perspective, but the kernel still considers the underlying block device in use.

The way to look across all mount namespaces is to inspect /proc/*/mountinfo directly:

1

sudo grep -l btrfs /proc/*/mountinfo

This searches the mountinfo file of every running process and returns the paths of those that reference btrfs. In my case, it returned:

1
2
3

/proc/5359/mountinfo
/proc/5440/mountinfo
/proc/5520/mountinfo

Checking those PIDs:

• PID 5359: tini -- /bin/bash -c start.sh
• PID 5440: docker container
• PID 5520: docker container API process

The containers had /dev/sdc mounted at /data inside their namespace. Pausing the containers leaves them running with their mount namespaces intact. The Btrfs device set was still held open by the kernel on behalf of those containers, which is why btrfs rescue could not obtain exclusive access to both RAID1 members.

The Fix

Bringing the stack down properly rather than just pausing it:

1

sudo docker compose down

Afterwards, re-running the grep returned nothing:

1
2

sudo grep -l btrfs /proc/*/mountinfo
(no output)

The filesystem was now genuinely unmounted from all namespaces. Retrying the cache clear:

1

sudo btrfs rescue clear-space-cache v1 /dev/sdc

1

Free space cache cleared

Mounting with v2 afterwards:

1

sudo mount -o space_cache=v2 /dev/sdc /mnt/btrfs

Confirmed via:

1
2

cat /sys/fs/btrfs/<UUID>/features/free_space_tree
1

And definitively via:

1

sudo btrfs inspect-internal dump-tree -t 10 /dev/sdc

which showed FREE_SPACE_INFO, FREE_SPACE_EXTENT and FREE_SPACE_BITMAP entries under a FREE_SPACE_TREE ROOT_ITEM — the v2 tree was created and active.

Takeaway

There are two things worth noting here. First, ERROR: cannot open device '/dev/sda': Device or resource busy is not a helpful error message when nothing on the host appears to be using the device. A message along the lines of "device held open in a non-host mount namespace" would save significant time. The existing tooling has a blind spot here.

Second, sudo grep -l btrfs /proc/*/mountinfo is a genuinely useful command in situations like this. It bypasses the limitations of fuser and lsof by going directly to the kernel's per-process mount information, which covers all namespaces. Worth keeping in your back pocket whenever a block device reports busy and nothing obvious explains why.