ExFAT

Public service announcement: never use ExFAT as the file format of the SSD or HDD you are editing on. ExFAT should only be used for transferring files between Windows and Mac, and should not be used for any important files. It is not a journaled file system (https://en.wikipedia.org/wiki/Journaling_file_system) which significantly reduces its file system robustness.

ExFAT has been adopted as the default file system for most SDXC cards, because it fits its use case the best: with a Maximum file size limit of 16EB (exabyte, which is 16 million TB) and is able to be read on both Windows and Mac. This is also the reason why many people say “SD cards are not reliable”, ExFAT is often the root of the problem. Trade-offs, flexibility for reliability. This is why you should never edit on an SD card directly from the cameras, and this is the reason why FCP insists on it.

Alert BEFORE the instructions: reformatting a drive will permanently erase everything on that drive completely, so ensure you have sufficient backup (on a drive or drive that is not formatted to ExFAT) before executing the following steps for reformatting:

Follow this Apple support document on how to use the Disk Utility application that comes installed with every Mac to reformat your drives to a better file system:https://support.apple.com/en-ca/guide/disk-utility/dskutl14079/mac follow the link in step 6 “choose a file system format” and read that article to find the best format for you. In short, APFS for MacOS 10.13 and later, Mac OS Extended for macOS 10.12 or earlier.

https://eshop.macsales.com/blog/80813-picking-the-right-drive-format/

When I made this post, I knew there was going to be at least one comment of this type. I have already addressed it in my original post, "good for you, and good luck", but since I have now been called "without any proof" and "mood creating", I feel compelled to make a response.

The choice of which file system to use is absolutely your choice. It is my personal opinion that ExFAT is unreliable based on what I have learned and read, and that's what pushed me to write this post in the first place.

No decent system is designed to be unreliable. All file systems are designed to work in a given environment and all file systems will fail under certain circumstances, the difference is the failure rate. It is entirely possible that a person can use any file system for an extended period of time without any problems, ExFAT included; that is what it is suppose to do. The difference is in the ability to maintain data integrity when encountering problems. Reading just this comment section with at least a dozen people explicitly claiming that they have had problems with ExFAT, I would consider that good enough "real life proof" that ExFAT has a relatively high chance of failure, and I really cannot afford to test my luck with ExFAT.

The reason I did not feel the need to provide “proof” is that it is very technical and complex, so I did not want to make this post even longer. But since some people prefer to know it, after consulting with one of the contributors to the official Microsoft ExFAT documentation, who is a current computer science professor at the University of Manitoba, I have compiled a list for anyone who is interested:

We must start from the official Microsoft ExFAT documentation

In which, the three design goals are

1. Retain the simplicity of FAT-based file systems.

2. Enable very large files and storage devices.

3. Incorporate extensibility for future innovation.

Notice that it does not mention anything regarding reliability, because it was never one of the designs objectives. This is a direct contrast to some of the other more modern file systems:

"Btrfs is a computer storage format that combines a file system based on the copy-on-write principle with a logical volume manager" (COW was designed to prioritize resiliency over speed)

"Ext4 is an advanced level of the ext3 filesystem which incorporates scalability and reliability enhancements for supporting large filesystems (64 bit) in keeping with increasing disk capacities and state-of-the-art feature requirements."

"NTFS uses its log file and checkpoint information to restore the consistency of the file system when the computer is restarted after a system failure."

Engineering is about compromise. If you want big and fast and simple, reliability must go.

One of the biggest problems is that ExFAT is derived from FAT which was organically designed in the 1970s, when storage size, speed and deployment environments were very different. One of the fundamental ways for adding reliability to modern file systems is through implementing a transaction log, or a journal. en.wikipedia.org/wiki/Journaling_file_system Since this was only developed in the 90s and ExFAT was aimed to retain the simplicity of FAT, this was not added as a structural improvement for ExFAT. Further more, circling back to the Microsoft documentation, you can notice that there is no concept of error detection in the form of checksums, which is another very important aspect of modern storage devices. Consumer flash storage devices are design to minimize cost, and in return some studies have found that the error rate of raw flash media operation on SSDs is somewhere between 1-10%, which is CRAZY high. Quoting another colleague's words, SSDs even working today “is a miracle of linear algebra”. Use a file system that supports some type of error correcting to help maintain this miracle. Modern operating systems all include lots of layers of error detecting and correcting, which are all designed to work in conjunction with file system level error detecting and correcting in the first place. Along with hardware and firmware level error correcting, you have the best chance in data resiliency.

If you really want to get into the programming side of things, to understand the exact technical implication behind what I have described in simple terms as best I can, https://home.cs.umanitoba.ca/~fbristow/ is where you need to go. All of his lectures on operating systems, including file systems design are posted online. There is really a lot to learn, so be patient and enjoy!

To summarize, ExFAT was designed and intended for flash media and removable media because it is flexible, fast and simple. But using it to transfer media implies that you already have a copy of the media somewhere else, so losing it is not a big deal. This is one of the other reasons why reliability was not considered necessary when they developed it from FAT. Use it for what it is designed for, transfer data on USB sticks. Do not use it for what it is not designed for, which is to “rely on it”, aka be plugged into a computer 24/7 and edit TBs of video and other important documents directly on it.