What is XFS/filesystem corruption and whose fault is it?

Sometimes OSDs fail to start. One reason this can happen is that the OSD's data store is in a partition that is formatted with an XFS filesystem, and this filesystem has been corrupted in some way.

Theoretically, XFS is "corruption-proof" but there are several preconditions that must be fulfilled before this can be relied upon:

1. disk controller must be in "write through" mode
2. alternatively, if the controller is in "write back" mode, it must be equipped with a Battery Back Up (BBU)
3. if relying on "write back" mode with BBU, the battery in BBU must be in good condition (and even if the BBU battery is in good condition, it can only preserve the filesystem journal for so long)
4. disk caches should be disabled
5. if using an SSD, it must be equipped with a "supercapacitor" to protect against loss of data in the SSD memory buffer (most "enterprise" SSDs have this)
6. the filesystem must not be mounted with the "nobarrier" mount option ("it's against the RADOS guarantees and expectations for an OSD to go back in time after committing an update and *can* result in failure").

NOTE: be sure to read Questions 24-28 of the XFS FAQ for details on write cache, write barrier, and disk controller settings!

If you experience a power outage or other crash, and XFS filesystems fail to mount afterwards, please double-check that these conditions were fulfilled before opening a bug against XFS or any systems, such as Ceph, that rely on XFS filesystem consistency.

Also, in any bug you open, provide detailed information about your disk controller, how it is configured, presence of BBU and state of the battery, how long the power was out, etc.

How to resize ext3/ext4 filesystem (including root)?

The other day I had to resize (i.e. shrink) the root filesystem on one of our machines. Today a colleague wrote the following concise HOWTO for the same:

1. Reduce the size the file system on your root partition with the tool
   applicable to the file system you have on that partition, such as
   resize2fs, to a size smaller than you need.

2. Resize the partition to the desired size (it has to be bigger than the
   file system from the previous step) using a tool such as fdisk.

3. Increase the file system size to completely fill the new partition
   size (again resize2fs, depending on your file system).

Working with the XFS filesystem

XFS is a modern filesystem suitable for data partitions.

General:
* RHEL Storage Administration Guide. Chapter 8. The XFS File System (practical info: create an XFS filesystem, grow it, quotas, backup, restore, etc.)
* SLES11-SP2 Storage Administration Guide -- section on XFS (technical overview; features)

Practical:
* Make sure to do 'zypper in xfsprogs' (you may have to install the 'filesystems' repository first)