New NTFS Driver in Kernel 5.15

Linux kernel release 5.15 introduced a new NTFS file system driver named NTFS3. This kernel driver was originally developed by Paragon Software as a commercial solution (more about the ntfs3 driver can be found on their FAQ page).

NTFS3 is not to be confused with NTFS-3G, a user space driver that employs the FUSE or “file system in user space” approach. There is a 3rd driver available – simply NTFS – that was shipped as the standard NTFS driver on Linux, but it lacked support for many of the Microsoft NTFS features (like writing to disk). Most of us dealing with NTFS drives have installed and use the NTFS-3G driver.

NTFS3 Read Performance

While I usually let others have fun with the early bugs that come along with new software releases, I felt I could give the NTFS3 driver a save trial in my new hash scripts. (I’ll probably post the scripts here soon.)

Depending on the storage media and size, hashing an entire file system can take a long time. For example hashing a 2.3TB file system on a HDD will take some 4 hours to finish.

Hashing files is a read operation. The hash program reads the entire file(s) and calculates the hash value. I've chosen the xxhash utility and the XXH128 hash since it is extremely fast while producing minimal collisions. From time to time the file hash is recalculated and compared against the stored hash to detect data corruption through storage / media failure, bit rot, you name it.

Since xxhash works at memory speed, any difference between using the NTFS3 kernel driver versus the NTFS-3G user space driver should be due to drive and driver performance.

xxhash Read Speed Test Using Different NTFS Drivers

In my first quick test I used a relatively small file system located on a Samsung SSD 970 EVO Plus 1TB NVMe drive with only 365 GB of data to hash. Below are the results of this read test.

NVMe storage

Hardware: Samsung SSD 970 EVO Plus 1TB NVMe drive
File system: Microsoft NTFS partition on LVM volume

NTFS-3G driver	NTFS3 kernel driver
Data hashed 393 GB	Data hashed 393 GB
real 4m38.437s	real 3m38.732s
user 0m23.902s	user 0m20.991s
sys 1m47.408s	sys 2m42.796s
1445 MB/s	1840 MB/s
100% of real time	73% of real time

First I created a snapshot of the LVM volume. Then I mapped the Windows partitions on the snapshot using the kpartx mapper. Finally I entered the following commands to mount the NTFS partition:

For the NTFS-3G driver:

mount -t ntfs -o ro,nls=utf8,dmask=027,fmask=137,uid=$user,gid=$group,windows_names /dev/mapper/vmvg-workdrive_snap /mnt/place-to-mount

Note: On my Manjaro Linux system, the -t ntfs option links to ntfs-3g:

/bin/mount.ntfs -> /usr/bin/ntfs-3g

For the NTFS3 kernel driver:

mount -t ntfs3 -o ro,iocharset=utf8,dmask=027,fmask=137,uid=$user,gid=$group /dev/mapper/vmvg-workdrive_snap /mnt/place-to-mount

As you can see, some of the -o options have changed, others like windows_names are gone!

The bottom line is that the new NTFS3 kernel improved read speed by 25% on a NVMe drive. Not bad.

RAID1 HDD Disks

Hardware: 2x Seagate 12TB Exos X16 7200 rpm SATA III 3.5″ Internal HDD
File system: Microsoft NTFS partition on LVM volume

NTFS-3G driver	NTFS3 kernel driver
Data hashed 2.3 TB	Data hashed 2.3 TB
real 195m31.009s	real 223m14.337s
user 2m31.048s	user 2m8.613s
sys 9m40.421s	sys 17m27.756s
200 MB/s	176 MB/s
100%	114%

In this benchmark the good old NTFS-3G driver came out as the winner! Not by a big margin, but still. More noticeable are the difference in system CPU time where the new kernel driver uses up almost twice the CPU time.

I will repeat the test to make sure there is no mistake. However, it looks like the NTFS-3G driver is the better choice for read-heavy workloads on conventional disks.

Write Speed Test

I have not done any write speed tests and probably never will. What’s the point in using a closed-source, proprietary file system for storage?

Using the New NTFS3 Driver

(Important note: On November 21, 2023 one of my NTFS drives showed corrupted directories. While unsure about the cause, it may have to do with the ntfs3 kernel driver. I wrote about it under “Does the Linux NTFS3 Driver Corrupt Directories?“.)

First of all you need kernel release 5.15 or newer. Depending on the distribution, NTFS3 may not be available in the first version(s) of that release. Under Manjaro Linux you need version 5.15.2 or newer. Check to see if NTFS3 is installed using:

[ -d "/lib/modules/$(uname -r)/kernel/fs/ntfs3" ] && echo "NTFS3 is here"

Once you verified that the new NTFS3 driver is available, you need to tell the system to use it. You have to do that manually for now, at least under Manjaro (see also the Arch Linux wiki).

Search the /etc/fstab file for NTFS entries:

$ grep -i ntfs /etc/fstab
/dev/disk/by-partlabel/wdbook /media/heiko/wdbook 	ntfs-3g  noauto,rw,user,nls=utf8,uid=1000,gid=1000,dmask=027,fmask=137,nofail,windows_names,locale=en_US.utf8  0 0

My fstab file holds only one entry for an external WDbook drive. In this case I’m using the “old” NTFS-3G driver as specified by -t ntfs-3g.

To enable the NTFS3 kernel driver, you need to edit your /etc/fstab file (as root user). But first make a backup of the file, for example like that:

sudo cp /etc/fstab /etc/fstab-${EPOCHSECONDS}

The above command produces a backup file with a unique number that can be easily sorted. Hint: the bottom of the list produced with ls -l /etc/fstab* will be the latest copy you created.

Before you change the driver in the /etc/fstab, test your mount options by mounting the drive or partition manually:

1. If the drive is mounted, unmount using umount /dev/device_file, for example: umount /dev/disk/by-partlabel/wdbook

2. Now manually mount the drive with the modified mount command. In my example that would be:

$ sudo mount -t ntfs3 -o rw,iocharset=utf8,uid=1000,gid=1000,dmask=027,fmask=137 /dev/disk/by-partlabel/wdbook /media/heiko/wdbook

Note the -t ntfs3 -o … command line options used with mount. Also note the change from nls=utf8 to iocharset=utf8. Some options like locale=en_US.utf8 are no longer available. Check out the syntax and options for the new NTFS3 driver.

3. Check that everything works as you expect. In particular, pay attention to the file permissions. The example above mounts the drive with user ID and group ID set to 1000. This is typically your user ID, if you are the primary or only user of the Linux machine (use the id command to find out).
   Setting the uid/gid allows you to edit/change/delete the content on that drive. My dmask and fmask entries specify sensible permissions for folders and files you are going to create – see here for a good explanation. Change them to fit your needs. If you don’t specify them, the process that creates the files or directories determines the permissions.

4. When done checking, unmount the drive (see above) and edit the /etc/fstab file. In my example I change the entry to:

/dev/disk/by-partlabel/wdbook /media/heiko/wdbook 	ntfs3  noauto,rw,user,iocharset=utf8,uid=1000,gid=1000,dmask=027,fmask=137,nofail  0 0

The above entry prevents the system from auto-mounting the drive on startup or when connecting it to a USB port. It allows the user (me) to manually mount the drive when needed.

For convenience I’m listing the file system independent basic options:

defaults – use default options: rw, suid, dev, exec, auto, nouser, and async.

noauto – do not mount when mount -a is given (e.g., at boot time)

user – allow a user to mount

owner – allow device owner to mount

comment or x- – for use by fstab-maintaining programs

nofail – do not report errors for this device if it does not exist.

5. When satisfied, save the fstab file. Then mount the drive manually using mount specifying only the mountpoint:

sudo mount /media/heiko/wdbook

If you added the “user” option to the fstab entry, you won’t need sudo:

mount /media/heiko/wdbook

You can then mount and unmount the drive using your favorite file browser.

6. Let’s make sure the drive is indeed mounted using the new NTFS3 kernel driver:

$ cat /proc/mounts | grep -i ntfs3
/dev/sdf1 /media/heiko/wdbook ntfs3 rw,nosuid,nodev,noexec,relatime,uid=1000,gid=1000,fmask=37777600137,dmask=37777600027,iocharset=utf8 0 0

Yep, it is!

Should You Switch to the New NTFS3 Driver?

Well, that depends. If all you care is read speed – like I do – you have to consider the storage media. In my test the new NTFS3 kernel driver gave a 27% boost with a NVMe flash drive when reading data. But it was 14% slower on a regular HDD.

Most “normal” users would probably not even notice a difference. If you just edit some document files on your NTFS drive from time to time there is probably no need (yet) to switch.

I use rsync in a script or Linux utility to frequently backup larger volumes of data to a network server or to external drives. A single RAW photo takes up 50-70 MB, when converted to tiff or psd it will use somewhere around 500 MB and more. 4K videos occupy many Gigabytes of storage.

My needs have changed with the implementation of the xxhash utility on all my drives. I have seen files go bad over time and the hash can identify file corruption. Updating the hashes after adding or modifying files takes perhaps a minute or two. But checking all the stored files against their hash takes several hours on an internal HDD.

The new NTFS3 kernel driver brings a minor improvement in disk read speed with NVMe drives. However, it performed worse with HDD drives when compared to the old and trusted NTFS-3G user space driver.

The test results should – as always – be taken with some caution, though. All my NTFS drives are residing on LVM volumes which may impact performance. Inside the Windows VM, using virtio drivers, the drives perform as expected and hold their ground compared to Windows on bare metal.

Before you switch horses or drivers in this case, take the time to benchmark your drives! For example, mounting a NTFS drive that resides on a physical partition may reveal different results. File size differences can have a huge impact too (in my case most files are more than 500K, mostly around 15-70MB per file). Last not least the computer hardware, free memory and particularly the workload – read, write, or both.

You should also keep a look at the development of the NTFS-3G driver. This Market Research Telecast sheds a little light behind the scenes. Phoronix tests show a very strong performance lead for the new NTFS3 kernel driver when used in their compile benchmark (I wonder who compiles on a Linux system using NTFS drives???).

TL;DR: Test the driver before switching!