diff --git a/content/ceph-benchmarking/index.md b/content/ceph-benchmarking/index.md index bab675a..eef25ef 100644 --- a/content/ceph-benchmarking/index.md +++ b/content/ceph-benchmarking/index.md @@ -3,16 +3,23 @@ title = "Ceph RBD Benchmarking" date = 2026-02-22 description = "My recent ceph rbd benchmarking adventure" +[extra] +toc = true + [taxonomies] categories = ["Homelab"] tags = ["Homelab", "Ceph"] +++ -## Motivation +Running a small set of benchmarks to find out what my ceph cluster is capable of. + + + +# Motivation I have been running a ceph cluster in my homelab for about 2 years now, but never properly benchmarked it, let alone wrote down my findings or any potential conclusions. I still don't know what kind of performance to expect or what would be considered good or expected for my setup, but having numbers without context is still better than no numbers at all. -## Benchmark Setup +# Benchmark Setup This covers all the steps I did to setup my benchmarking, so anyone could follow along and for me to reference later to repeat benchmarks properly. {% details(summary="Create Benchmark User in Ceph") %} @@ -49,7 +56,7 @@ Setup the benchmark itself: 5. `mount /dev/rbd0 /mnt/bench` {% end %} -## Benchmarks +# Benchmarks All benchmarks are run with the same configuration, only changing the access patterns (read/write, random/sequential). Key configuration options are: - using libaio @@ -101,7 +108,7 @@ bs=4M ``` {% end %} -## Results +# Results {% details(summary="Random Reads") %} {{ fio_benchmark(path="content/ceph-benchmarking/benchmarks/random_read.json") }} {% end %} @@ -118,7 +125,7 @@ bs=4M {{ fio_benchmark(path="content/ceph-benchmarking/benchmarks/seq_write.json") }} {% end %} -## Conclusion +# Conclusion 1. Overall I am satisfied with the performance of the cluster for my current use-case 2. There is a lot of room for improvement in the low queue-depth range 3. The network is not really a limiting factor currently @@ -126,7 +133,7 @@ bs=4M - My client used for testing was limited by the network, evident by the fact that the highest speed achieved is ~1.2GB/s (~10Gb/s) 4. My smallest node (the embedded epyc) could be the limiting factor as in some benchmarks, it reached 100% cpu usage, while my other nodes never exceeded 40% -## Extra Details +# Extra Details {% details(summary="Cluster Hardware") %} - 10 GbE Networking between all nodes - Node @@ -175,7 +182,7 @@ jq '[.jobs[] | { iodepth: ."job options".iodepth, bs: ."job options".bs, operati ``` {% end %} -## Future Work +# Future Work - Try directly on the block device - Try this using xfs instead of ext4 - Try this with and without drive caches