一 前期準(zhǔn)備

1.1 前置條件

1.2 準(zhǔn)備工作

1.3 其他注意事項(xiàng)

附一：組件端口說明

二 部署相關(guān)知識(shí)點(diǎn)

2.1 Ansible介紹

2.2 Ansible部署Ceph相關(guān)yml

2.3 yml主要相關(guān)參數(shù)

2.4 客戶端相關(guān)命令

2.5 對(duì)象object相關(guān)命令

三 正式部署

3.1 部署節(jié)點(diǎn)配置主機(jī)名

[root@servera ~]# vi /etc/hosts  172.25.250.10 servera  172.25.250.12 serverc  172.25.250.13 serverd  172.25.250.14 servere

3.2 創(chuàng)建相關(guān)用戶

[root@servera ~]# useradd student
[root@servera ~]# echo student | passwd –stdin student  #創(chuàng)建非root的管理用戶
[root@servera ~]# for i in {a..e}; do echo “====server${i}====”;ssh root@server${i} ‘useradd -d /home/student -m student; echo “student” | passwd –stdin student’; done                #所有OSD server節(jié)點(diǎn)創(chuàng)建student用戶
[root@servera ~]# for i in {a..e}; do echo “====server${i}====”;ssh root@server${i} ‘useradd -d /home/ceph -m ceph; echo “redhat” | passwd –stdin ceph’; done
[root@servera ~]# for i in {a..e}; do echo “====server${i}====”;ssh root@server${i} ‘echo “student ALL = (root) NOPASSWD:ALL” > /etc/sudoers’; done
[root@servera ~]# for i in {a..e}; do echo “====server${i}====”;ssh root@server${i} ‘chmod 0440 /etc/sudoers’; done

3.3 配置部署節(jié)點(diǎn)免密鑰

[root@servera ~]# su - student  [student@servera ~]$ ssh-keygen -f ~/.ssh/id_rsa -N ''  [student@servera ~]$ for i in {a..e}; do echo "====server${i}====";ssh-copy-id student@server$i;ssh-copy-id ceph@server$i; done

3.4 配置Ansible Inventory

[student@servera ~]$ sudo vi /usr/share/ceph-ansible/ansible.cfg  log_path = /tmp/ansible.log          #修改日志路徑為student用戶可寫入的/tmp路徑  deprecation_warnings = False  #禁用在ansible-playbook輸出結(jié)果相關(guān)必須要警告

[student@servera ~]$ sudo vi /etc/ansible/hosts
[mons]
server[c:e]

[mgrs]
server[c:e]
[student@servera ~]$ ansible mons -m ping  #測試mons組節(jié)點(diǎn)通信
[student@servera ~]$ ansible mgrs -m ping  #測試mgrs組節(jié)點(diǎn)通信
[student@servera ~]$ ansible mons -m command -a id #通過命令測試mons組節(jié)點(diǎn)
[student@servera ~]$ ansible mgrs -m command -a id #通過命令測試mgrs組節(jié)點(diǎn)

3.5 創(chuàng)建site.yml

[student@servera ~]$ cd /usr/share/ceph-ansible/
[student@servera ceph-ansible]$ sudo cp site.yml.sample site.yml
[student@servera ceph-ansible]$ sudo vi site.yml
#……
– hosts: osds
  gather_facts: false
  become: True
  serial: 1                  #在osd（80行左右）添加此行

3.6 創(chuàng)建all.yml

[student@servera ~]$ cd /usr/share/ceph-ansible/group_vars/
[student@servera group_vars]$ sudo cp all.yml.sample all.yml
[student@servera group_vars]$ sudo vi all.yml
—
dummy:
ntp_service_enabled: false   #本實(shí)驗(yàn)采用chrony進(jìn)行時(shí)鐘同步
ceph_origin: repository
ceph_repository: rhcs
ceph_rhcs_version: “3”
ceph_repository_type: cdn
rbd_cache: “true”    #開啟RBD回寫緩存
rbd_cache_writethrough_until_flush: “false” #在切換回寫之前，不從寫透開始。
rbd_client_directories: false  #不要?jiǎng)?chuàng)建客戶機(jī)目錄(它們應(yīng)該已經(jīng)存在)。
monitor_interface: eth0
journal_size: 1024    #本環(huán)境存儲(chǔ)設(shè)備很小，OSD日志比通常建議的要小
public_network: 172.25.250.0/24
cluster_network: “{{ public_network }}”
ceph_conf_overrides:
  global:
    mon_osd_allow_primary_affinity: 1
    mon_clock_drift_allowed: 0.5  #允許MON時(shí)鐘間隔最多0.5秒
    osd_pool_default_size: 2
    osd_pool_default_min_size: 1  #降低存儲(chǔ)池復(fù)制大小的默認(rèn)設(shè)置
    mon_pg_warn_min_per_osd: 0  #見提示一
    mon_pg_warn_max_per_osd: 0  #見提示二
    mon_pg_warn_max_object_skew: 0  #見提示三
  client:
    rbd_default_features: 1   #僅為以后的練習(xí)啟用一組特定的客戶機(jī)功能

3.7 正式部署Ceph集群

[student@servera ~]$ cd /usr/share/ceph-ansible/  [student@servera ceph-ansible]$ ansible-playbook site.yml

3.8 確認(rèn)驗(yàn)證Ceph集群

[student@servera ~]$ ssh ceph@serverc ceph -s

[student@servera ~]$ ssh ceph@serverc cat /etc/ceph/ceph.conf

[student@servera ~]$ ssh ceph@serverc ps aux | grep ceph-mon

3.9 創(chuàng)建osds.yml

[student@servera ~]$ cd /usr/share/ceph-ansible/group_vars/
[student@servera group_vars]$ sudo cp osds.yml.sample osds.yml
[student@servera group_vars]$ sudo vi osds.yml
—
dummy:
osd_scenario: “collocated”   #OSD使用并列的OSD形式
devices:
  – /dev/vdb    #使用/dev/vdb作為后端存儲(chǔ)設(shè)備

3.10 配置Ansible Inventory

[student@servera ~]$ sudo vi /etc/ansible/hosts
[mons]
server[c:e]

[mgrs]
server[c:e]

[osds]
server[c:e]     #追加osds組

3.11 正式部署OSD節(jié)點(diǎn)

[student@servera ~]$ cd /usr/share/ceph-ansible/  [student@servera ceph-ansible]$ ansible-playbook site.yml

3.12 確認(rèn)驗(yàn)證OSD節(jié)點(diǎn)

[student@servera ~]$ ssh ceph@serverc ceph -s

[student@servera ~]$ ssh ceph@serverc ceph -w  #使用此命令監(jiān)視集群事件

3.13 測試及驗(yàn)證

[ceph@serverc ~]$ sudo systemctl stop ceph-mon.target #停止serverc的mon進(jìn)程  [ceph@serverc ~]$ ceph -s            #觀察Ceph狀態(tài)

[ceph@serverc ~]$ sudo systemctl start ceph-mon.target #重啟開啟mon進(jìn)程  [ceph@serverc ~]$ sudo systemctl stop ceph-osd.target #停止serverc的osd進(jìn)程  [ceph@serverc ~]$ ceph -s            #觀察Ceph狀態(tài)

[ceph@serverc ~]$ ceph osd tree    #查看osd情況

[ceph@serverc ~]$ sudo systemctl start ceph-osd.target #重啟開啟osd進(jìn)程  [ceph@serverc ~]$ sudo systemctl stop ceph-osd@0         #停止serverc的osd id為0的進(jìn)程  [ceph@serverc ~]$ ceph osd tree    #查看osd情況

[ceph@serverc ~]$ sudo systemctl start ceph-osd@0 #重啟開啟osd 0進(jìn)程  [ceph@serverc ~]$ ceph -s    #觀察Ceph狀態(tài)

[ceph@serverc ~]$ ceph -v   #查看Ceph版本

3.14 創(chuàng)建client.yml

[student@servera ~]$ cd /usr/share/ceph-ansible/group_vars/
[student@servera group_vars]$ sudo cp clients.yml.sample clients.yml
[student@servera group_vars]$ sudo vi clients.yml
—
dummy:
copy_admin_key: true

3.15 配置Ansible Inventory

[student@servera ~]$ sudo vi /etc/ansible/hosts
[mons]
server[c:e]

[mgrs]
server[c:e]

[osds]
server[c:e]

[clients]
servera     #追加client客戶端

3.16 正式部署client節(jié)點(diǎn)

[student@servera ~]$ cd /usr/share/ceph-ansible/  [student@servera ceph-ansible]$ ansible-playbook site.yml

3.17 確認(rèn)驗(yàn)證

[kiosk@foundation0 ~]$ ssh ceph@servera   #使用ceph用戶登錄servera  [ceph@servera ~]$ ceph -s     #查看Ceph集群

四 擴(kuò)容Ceph集群

4.1 擴(kuò)容前置條件

4.2 創(chuàng)建相關(guān)用戶

[root@serverf ~]# useradd student
[root@serverf ~]# echo student | passwd –stdin student          #創(chuàng)建非root的管理用戶
[root@serverf ~]# useradd -d /home/student -m student; echo “student” | passwd –stdin student’                                                #所有OSD server節(jié)點(diǎn)創(chuàng)建student用戶
[root@serverf ~]# useradd -d /home/ceph -m ceph; echo “redhat” | passwd –stdin ceph’
[root@serverf ~]# echo “student ALL = (root) NOPASSWD:ALL” > /etc/sudoers’
[root@serverf ~]# chmod 0440 /etc/sudoers

4.3 配置部署節(jié)點(diǎn)免密鑰

[root@servera ~]# su - student  [student@servera ~]$ ssh-copy-id student@serverf;ssh-copy-id ceph@serverf

4.4 擴(kuò)容額外的OSD主機(jī)

[student@servera ~]$ sudo vi /etc/ansible/hosts  #配置Ansible Inventory
[student@servera ~]$ sudo vi /etc/ansible/hosts
[mons]
server[c:e]

[mgrs]
server[c:e]

[osds]
server[c:e]
serverf       #追加serverf

[clients]
servera

4.5 添加額外OSD存儲(chǔ)設(shè)備

devices:    - /dev/vdb    - /dev/vdc    - /dev/vdd      #追加存儲(chǔ)設(shè)備

4.6 正式部署OSD節(jié)點(diǎn)

[student@servera ~]$ cd /usr/share/ceph-ansible/  [student@servera ceph-ansible]$ ansible-playbook site.yml

4.7 確認(rèn)驗(yàn)證

[ceph@servera ~]$ ceph -s

 [ceph@servera ~]$ ceph osd tree

[ceph@servera ~]$ ceph osd df