Showing posts with label ceph. Show all posts
Showing posts with label ceph. Show all posts

2020-10-30

CephFS cheatsheet

This page assumes you have a Ceph cluster running, and have created a CephFS there. Now, you want to actually use the CephFS. All of the actions described below take place on the client - i.e., computer where you want to use CephFS.

Mount CephFS using the kernel client

First, install ceph-common (any version) to get the mount.ceph helper binary.

Second, copy over /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring from one of the Ceph cluster nodes.

Third, make a mountpoint - for example:

# mkdir -p /mnt/cephfs

Fourth, find out the hostname or IP address of one of the MON nodes of your Ceph cluster.

You can now mount CephFS by doing (replace MON_NODE with the hostname or IP address of the MON node):

# mount -t ceph MON_NODE:/ /mnt/cephfs -o name=admin

2020-09-10

How to shut down a Ceph cluster

Shutting down a Ceph cluster is not just a matter of running poweroff on each node. If you do that, the cluster will get confused when you start to bring it up. The MONs will see only a subset of the OSDs, and will start marking the missing ones down and PGs will go into degraded state. This is because the cluster cannot by itself distinguish between a node that has been powered down and one that has been vaporized, or burnt to a crisp (for example).

It is of course possible to do a controlled shutdown of all the nodes and then bring the cluster back up without any bad effects. But the following procedure might not be it! Don't assume that it will work for your cluster!


Shutdown procedure

I don't advise anyone to use this procedure, because it might cause loss of precious data. Use the procedure AT YOUR OWN RISK!

  1. Stop the clients from using your cluster (this step is only necessary if you want to shutdown your whole cluster)
  2. Issue the command ceph -s and verify that the cluster is HEALTH_OK status
  3. ceph osd set noout
  4. ceph osd set nobackfill
  5. ceph osd set norecover
  6. On each node: systemctl stop ceph.target
  7. poweroff

Bring the cluster back up

I don't advise anyone to use this procedure, because it might cause loss of precious data. Use the procedure AT YOUR OWN RISK!

Once the cluster has been powered up, the OSD flags that were set to protect the cluster from itself while it is being powered down and back up again must be unset in reverse order:

  1. ceph osd unset norecover
  2. ceph osd unset nobackfill
  3. ceph osd unset noout

2020-03-28

How to verify that an encrypted OSD is really encrypted?

Source: https://github.com/SUSE/doc-ses/issues/218

Since decrypting the data on an encrypted OSD disk requires knowledge of the corresponding dm-crypt secret key, OSD encryption provides protection for cases when a disk drive that was used as an OSD is decommissioned, lost, or stolen.

Scenario: You have deployed some OSDs that are supposedly encrypted, but don't know how to verify that.

First, run ceph-volume lvm list on the machine where the OSDs are running. The output might look something like this:

# ceph-volume lvm list

====== osd.3 =======

  [block]       /dev/ceph-d9f09cf7-a2a4-4ddc-b5ab-b1fa4096f713/osd-data-71f62502-4c85-4944-9860-312241d41bb7

      block device              /dev/ceph-d9f09cf7-a2a4-4ddc-b5ab-b1fa4096f713/osd-data-71f62502-4c85-4944-9860-312241d41bb7
      block uuid                m5F10p-tUeo-6ZGP-UjxJ-X3cd-Ec5B-dNGXvG
      cephx lockbox secret      
      cluster fsid              413d9116-e4f6-4211-a53b-89aa219f1cf2
      cluster name              ceph
      crush device class        None
      encrypted                 0
      osd fsid                  f8596bf7-000f-4186-9378-170b782359dc
      osd id                    3
      type                      block
      vdo                       0
      devices                   /dev/vdb

====== osd.7 =======

  [block]       /dev/ceph-38914e8d-f512-44a7-bbee-3c20a684753d/osd-data-0f385f9e-ce5c-45b9-917d-7f8c08537987

      block device              /dev/ceph-38914e8d-f512-44a7-bbee-3c20a684753d/osd-data-0f385f9e-ce5c-45b9-917d-7f8c08537987
      block uuid                1y3qcS-ZG01-Y7Z1-B3Kv-PLr6-jbm6-8B79g6
      cephx lockbox secret      
      cluster fsid              413d9116-e4f6-4211-a53b-89aa219f1cf2
      cluster name              ceph
      crush device class        None
      encrypted                 0
      osd fsid                  0f9a8002-4c81-4f5f-93a6-255252cac2c4
      osd id                    7
      type                      block
      vdo                       0
      devices                   /dev/vdc

Note the line that says encrypted 0 - this means the OSD is not encrypted. Here are the possible values:

      encrypted                 0  <- not encrypted
      encrypted                 1  <- encrypted
If you get the following error, it means the node where you are running the command does not have any OSDs on it:
# ceph-volume lvm list
No valid Ceph lvm devices found
Now, imagine that, in your case, ceph-volume lvm list shows encrypted 1 for a given OSD, so you know ceph-volume thinks that OSD is encrypted. How to gain more confidence that it really is encrypted? Ceph OSD encryption-at-rest relies on the Linux kernel's dm-crypt subsystem and the Linux Unified Key Setup ("LUKS"). When creating an encrypted OSD, ceph-volume creates an encrypted logical volume and saves the corresponding dm-crypt secret key in the Ceph Monitor data store. When the OSD is to be started, ceph-volume ensures the device is mounted, retrieves the dm-crypt secret key from the Ceph Monitors, and decrypts the underlying device. This creates a new device, containing the unencrypted data, and this is the device the Ceph OSD daemon is started on. Since the OSD itself does not know whether the underlying logical volume is encrypted or not, there is no ceph osd command that will return this information. But it is possible to query LUKS for it, as follows. First, get the device of the OSD logical volume you are interested in. This can be obtained from the ceph-volume lvm list output:
block device              /dev/ceph-d9f09cf7-a2a4-4ddc-b5ab-b1fa4096f713/osd-data-71f62502-4c85-4944-9860-312241d41bb7
Then, dump the LUKS header from that device:
# cryptsetup luksDump OSD_BLOCK_DEVICE
Here is what the output looks like when the OSD is NOT encrypted:
Device /dev/ceph-38914e8d-f512-44a7-bbee-3c20a684753d/osd-data-0f385f9e-ce5c-45b9-917d-7f8c08537987 is not a valid LUKS device.
And when it IS encrypted:
master:~ # cryptsetup luksDump /dev/ceph-1ce61157-81be-427d-83ad-7337f05d8514/osd-data-89230c92-3ace-4685-97ff-6fa059cef63a
LUKS header information for /dev/ceph-1ce61157-81be-427d-83ad-7337f05d8514/osd-data-89230c92-3ace-4685-97ff-6fa059cef63a

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha256
Payload offset: 4096
MK bits:        256
MK digest:      e9 41 85 f1 1b a3 54 e2 48 6a dc c2 50 26 a5 3b 79 b0 f2 2e 
MK salt:        4c 8c 9d 1f 72 1a 88 6c 06 88 04 72 81 7b e4 bb 
                b1 70 e1 c2 7c c5 3b 30 6d f7 c8 9c 7c ca 22 7d 
MK iterations:  118940
UUID:           7675f03b-58e3-47f2-85fc-3bafcf1e589f

Key Slot 0: ENABLED
        Iterations:             1906500
        Salt:                   8f 1f 7f f4 eb 30 5a 22 a5 b4 14 07 cc da dc 48 
                                b5 e9 87 ef 3b 9b 24 72 59 ea 1a 0a ec 61 e6 42 
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

2019-11-29

Use ceph-fuse to mount CephFS

The upstream documentation:

* nautilus - https://docs.ceph.com/docs/nautilus/cephfs/fuse/
* master - https://docs.ceph.com/docs/master/cephfs/fuse/

Steps to assert ceph-fuse is working


First, deploy a cluster with at least one MDS running.

Second, decide if you will run ceph-fuse as root or as a normal user and decide which machine will be the "client machine" (it must not be a part of the Ceph cluster).

Third, on the client machine, install /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring by copying these files from a MON node. If using non-root user, make sure both files have permissions 644 and the containing directory has permissions 755.

Fourth, on the client machine, install the ceph-fuse package.

Fifth, if using a normal user, make sure that user exists on the client machine and is a member of the ceph group.

Sixth, if using a normal user, set 'user_allow_other' in /etc/fuse.conf.

Seventh, get the IP address of a MON node by looking inside /etc/ceph.conf:

$ grep mon_host /etc/ceph/ceph.conf
mon_host = 10.20.24.200

Eighth, create a mount point:

$ mkdir ~/cephfs
$ ls ~/cephfs
$

Ninth, mount cephfs using the IP address of the MON found in step 6:

$ ceph-fuse -m 10.20.24.200 ~/cephfs
2019-11-29 13:51:13.688 7fafe9757040 -1 init, newargv = 0x5581396d32c0 newargc=7
ceph-fuse[27795]: starting ceph client
ceph-fuse[27795]: starting fuse
$

Tenth, when done, unmount cephfs:

$ fusermount -u ~/cephfs

2019-03-13

Run ceph CLI commands from Python

"mon_command" is a way to run ceph CLI commands via the librados Python API.

Which commands can I run?


https://github.com/ceph/ceph/blob/master/src/mon/MonCommands.h
https://github.com/ceph/ceph/blob/master/src/mgr/MgrCommands.h

Do you have a sample Python script?


#!/usr/bin/python3
import json
import rados

def run_command(cluster_handle, cmd):
    return cluster_handle.mon_command(json.dumps(cmd), b'', timeout=5)

cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
cluster.connect()
print(run_command(cluster, {"prefix": "osd safe-to-destroy", "ids": ["2"], "format": "json"}))
print(run_command(cluster, {"prefix": "osd ok-to-stop", "ids": ["2"], "format": "json"}))

2017-08-07

cephfs: ideal PG ratio between metadata and data pools

Scenario: I'm deploying CephFS for the first time. I know I need a metadata pool as well as a data pool, but I don't know how many PGs for one pool and the other? Can I use the same number, or should they be different?

This question is a FAQ. The number of PGs should be chosen in line with the expected amount of data. In a typical cephfs usage pattern, there will be less data in the metadata pool than in the data pool, but nobody really knows exactly how *much* less. In this thread, John Spray suggests "something like" a 1:4 metadata-to-data ratio.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-March/016965.html

Quoting John's answer there:
As you suggest, it is probably sensible to use a smaller number of PGs
for the metadata pool than for the data pool.  I would be tempted to
try something like an 80:20 ratio perhaps, that's just off the top of
my head.

Remember that there is no special prioritisation for metadata traffic
over data traffic on the OSDs, so if you're mixing them together on
the same drives, then you may see MDS slowdown if your clients
saturate the system with data writes.  The alternative is to dedicate
some SSD OSDs for metadata.

What is the optimum number of PGs for my pool?

Each pool has a fixed number of Placement Groups (PGs) *and* a fixed number of
OSDs. The former is set when the pool is created and the latter is determined
by the CRUSH map.

The number of PGs in a pool can be increased, but it cannot easily be decreased
(there is always the option to create a new pool, copy the data over, and
destroy the old pool).

If there are too few PGs per OSD, or too many, cluster performance will suffer.
In general, the number should be between 30 and 300 and the oft-quoted "rule of
thumb" for ideal performance is _approximately_ 100 PGs per OSD.

Since any given OSD may of course belong to several pools, depending on the
cluster topography and the CRUSH map, it's not easy to generalize about what
is the optimal/right number of PGs for an arbitrary pool. Still, the
Ceph PGs per Pool Calculator can help.

2016-11-25

How to increase debug levels and harvest a detailed OSD log

Your OSD doesn't start and you want to find out why. Here's how to increase the debug levels and harvest a detailed OSD log:

First, rotate the OSD log, or just do "cd /var/log/ceph ; mv ceph-osd.0.log ceph-osd.0.log-foo"

Then, edit /etc/ceph/ceph.conf to add the following lines to the [osd] section:

[osd]
    debug osd = 20
    debug filestore = 20
    debug ms = 1

Make sure you know the "number" (id) of the OSD. You can find this out by looking at, e.g. ceph osd tree

Then start the OSD manually, by running the following command as root:

# /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

Harvest the log - it is in /var/log/ceph/ceph-osd.${OSD_ID}.log

Finally, restore /etc/ceph/ceph.conf back the way it was! (Otherwise, the increased debug levels cause OSDs to emit huge quantities of log messages which could easily fill up your root filesystem.)

How to repair a leveldb database

Repairing a corrupted leveldb database turns out to be simple, but there is no guarantee that the database state after repair will be the same as it was before the corruption occurred!

First, install the leveldb Python module, e.g., using pip.

Then, determine the directory path where your leveldb database is stored. For example, the omap of a Ceph (hammer, jewel) OSD is generally stored in /var/lib/ceph/osd/ceph-$ID/current/omap

Finally, run the Python shell and enter the following commands:

$ python
...
>>> import leveldb
>>> leveldb.RepairDB('')
>>>

That's all.

2016-09-22

How to measure network throughput in Ceph cluster

According to the Ceph documentation, "the journal size should be at least twice the product of the "expected throughput" multiplied by filestore max sync interval" [1] where "expected throughput" is the lower of drive throughput and network throughput. The same documentation gives guidance that 7200 RPM spinners have approximate throughput of 100MB/sec, so we have a value for drive throughput. But what about network throughput? How to measure it?

iperf to the rescue


Measuring throughput is incredibly easy with iperf3.

For openSUSE or SUSE Linux Enterprise (SLE), packages are available in the network:utilities project in OBS - https://software.opensuse.org/download.html?project=network%3Autilities&package=iperf

Once you have the package installed, running a throughput test between two nodes (node1 and node2, hypothetically) is as easy as:

On node1:

# iperf3 -s

On node2:

# iperf3 -c $IP_ADDR_OF_NODE1


[1] http://docs.ceph.com/docs/jewel/rados/configuration/osd-config-ref/

2016-09-20

Single-node Ceph cluster

Sometimes I need a Ceph cluster for testing and I need it to be as simple as possible. Fortunately, Ceph can be convinced to run, self-contained, on a single machine. Of course, such a cluster should only be used for testing purposes, since it provides no failure resistance.

Single-node cluster requires some special configuration, but not too complicated. First, there can only be one MON in such a cluster. Second, the default "CRUSH chooseleaf" must be set to 0. Third, "size" and "min_size" pool parameters must be set appropriately, depending on the number of OSDs in the cluster.

Example ceph.conf for single-node cluster with two OSDs:

[global]
fsid = 44af0c2b-0423-4823-86f5-b21b5ed782b0
mon_initial_members = vanguard2
mon_host = 10.100.12.15
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_crush_chooseleaf_type = 0
osd_pool_default_size = 1
osd_pool_default_min_size = 1

For a single OSD, change the last line to osd_pool_default_min_size = 0. Ceph wants to provide failure resistance and will normally go into HEALTH_WARN if the cluster is not configured to sustain the failure of at least one OSD without data loss.