OpenStack 提供 Block Storage Service 的是 Cinder,其具体功能是:
-
提供 REST API 使用户能够查询和管理 volume、volume snapshot 以及 volume type
-
提供 scheduler 调度 volume 创建请求,合理优化存储资源的分配
-
通过 driver 架构支持多种 back-end(后端)存储方式,包括 LVM,NFS,Ceph 和其他诸如 EMC、IBM 等商业存储产品和方案
Cinder 架构
下图是 cinder 的逻辑架构图
Cinder 包含如下几个组件:
cinder-api
接收 API 请求,调用 cinder-volume 执行操作。cinder-volume
管理 volume 的服务,与 volume provider 协调工作,管理 volume 的生命周期。运行 cinder-volume 服务的节点被称作为存储节点。cinder-scheduler
scheduler 通过调度算法选择最合适的存储节点创建 volume。volume provider
数据的存储设备,为 volume 提供物理存储空间。 cinder-volume 支持多种 volume provider,每种 volume provider 通过自己的 driver 与cinder-volume 协调工作。Message Queue
Cinder 各个子服务通过消息队列实现进程间通信和相互协作。因为有了消息队列,子服务之间实现了解耦,这种松散的结构也是分布式系统的重要特征。Database Cinder 有一些数据需要存放到数据库中,一般使用 MySQL。数据库是安装在控制节点上的,比如在我们的实验环境中,可以访问名称为“cinder”的数据库。
在控制节点可以看到,部署了cinder-api,cinder-schedule,cinder-volume,控制节点即是存储节点,又是控制节点。当然也可以用单独的机器来部署cinder-volume
[root@cc07 ~]# ps -e|grep cinder 7007 ? 04:39:10 cinder-volume 7193 ? 10:36:20 cinder-volume13007 pts/6 04:27:02 cinder-api13385 pts/14 11:13:49 cinder-schedule13794 ? 04:41:05 cinder-volume14423 pts/28 00:24:10 cinder-backup19069 pts/6 10:17:50 cinder-api19070 pts/6 10:29:14 cinder-api19071 pts/6 10:25:13 cinder-api19072 pts/6 10:22:08 cinder-api19073 pts/6 10:31:14 cinder-api19074 pts/6 10:17:53 cinder-api19075 pts/6 10:29:18 cinder-api19076 pts/6 10:21:52 cinder-api19262 ? 10:58:48 cinder-volume19267 ? 10:57:51 cinder-volume19272 ? 10:55:53 cinder-volume
存储节点支持多种 volume provider,包括 LVM, NFS, Ceph, GlusterFS,以及 EMC, IBM 等商业存储系统。 cinder-volume 为这些 volume provider 定义了统一的 driver 接口,volume provider 只需要实现这些接口,就可以 driver 的形式即插即用到 OpenStack 中。下面是 cinder driver 的架构示意图:
在 cinder-volume 的配置文件 /etc/cinder/cinder.conf 中 volume_driver 配置项设置该存储节点使用哪种 volume provider 的 driver,下面的示例表示使用的是 LVM和ceph
[DEFAULT]enabled_backends=ceph,ceph-ssd,cephSsddebug = Truelog_dir = /var/log/kolla/cinderuse_forwarded_for = trueuse_stderr = Falseenable_v1_api = falsevolume_name_template = volume-%sglance_api_servers = http://172.16.12.7:9292glance_api_version = 2os_region_name = RegionOneosapi_volume_listen = 172.16.12.7osapi_volume_listen_port = 8776api_paste_config = /etc/cinder/api-paste.ininova_catalog_info = compute:nova:internalURLauth_strategy = keystone[database]connection = mysql+pymysql://cinder:cinder@172.16.12.7/cindermax_retries = -1[keystone_authtoken]auth_uri = http://172.16.12.7:5000auth_url = http://172.16.12.7:35357auth_type = passwordproject_domain_id = defaultuser_domain_id = defaultproject_name = serviceusername = cinderpassword = cindermemcache_security_strategy = ENCRYPTmemcache_secret_key = dKcXVdSMqdHcQUgjE0ZiB2L9VeGyhB49cSXsA8Ibmemcached_servers = 172.16.12.7:11211[oslo_concurrency]lock_path = /var/lib/cinder/tmp[oslo_messaging_rabbit]rabbit_userid = openstackrabbit_password = huacloudhuacloudrabbit_ha_queues = truerabbit_hosts = 172.16.12.7:5672[lvm]volume_group = cinder-volumesvolume_driver = cinder.volume.drivers.lvm.LVMVolumeDrivervolume_backend_name = iscsiiscsi_helper = lioadmiscsi_protocol = iscsi[ceph]volume_driver=cinder.volume.drivers.rbd.RBDDriverrbd_pool=volumesrbd_ceph_conf=/etc/ceph/ceph.confrbd_flatten_volume_from_snapshot=falserbd_max_clone_depth=5glance_api_version=1volume_backend_name=ceph[ceph-ssd]volume_driver=cinder.volume.drivers.rbd.RBDDriverrbd_pool=volumesrbd_ceph_conf=/etc/ceph/ceph.confrbd_flatten_volume_from_snapshot=falserbd_max_clone_depth=5glance_api_version=1volume_backend_name=ceph-ssd[cephSsd]volume_driver=cinder.volume.drivers.rbd.RBDDriverrbd_pool=volumesrbd_ceph_conf=/etc/ceph/ceph.confrbd_flatten_volume_from_snapshot=falserbd_max_clone_depth=5glance_api_version=1volume_backend_name=cephSsd
上面的lvm中volume_group指的是LVM中用vgcreate创建出来的卷组vg
基本命令
Volume Type
Cinder 允许用户创建 Volume 时通过 Volume Type 指定需要的 Capabilities,通过 Volume Type 的 Extra Specs 定义 Capabilities,cinder-volume 会在自己的配置文件 /etc/cinder/cinder.conf 中设置“volume_backend_name”这个参数,其作用是为存储节点的 Volume Provider 命名。这样,CapabilitiesFilter 就可以通过 Volume Type 的“volume_backend_name”筛选出指定的 Volume Provider。
主要命令
extra-specs-list Lists current volume types and extra specs. qos-associate Associates qos specs with specified volume type. qos-disassociate Disassociates qos specs from specified volume type. retype Changes the volume type for a volume. type-access-add Adds volume type access for the given project. type-access-list Print access information about the given volume type. type-access-remove Removes volume type access for the given project. type-create Creates a volume type. type-default List the default volume type. type-delete Deletes a volume type. type-key Sets or unsets extra_spec for a volume type. type-list Lists available 'volume types'. (Admin only will see private types) type-show Show volume type details. type-update Updates volume type name, description, and/or
△显示存储卷类型列表
(nova-api)[root@cc07 /]# cinder extra-specs-list+--------------------------------------+----------+-------------------------------------+| ID | Name | extra_specs |+--------------------------------------+----------+-------------------------------------+| 0a220534-3404-40ff-b0a0-a449429e13c1 | ceph | { 'volume_backend_name': 'ceph'} || 1bcf273a-79de-4847-9dce-57628b2cd185 | vmdk | { 'volume_backend_name': 'vmdk'} || 747632fd-e8f2-403a-978d-892002498497 | ceph_new | { 'volume_backend_name': 'ceph_new'} || 88d1db93-1104-489c-ac9c-389a4e4736a1 | ceph-ssd | { 'volume_backend_name': 'ceph-ssd'} || 9b59e61e-b588-43fc-9c47-16337b595e74 | iscsi | { 'volume_backend_name': 'iscsi'} || b1c40afc-a8e3-4b78-b534-5d73a8ba98a8 | cephSsd | { 'volume_backend_name': 'cephSsd'} |+--------------------------------------+----------+-------------------------------------+
△创建存储卷类型
(nova-api)[root@cc07 /]# cinder type-create --is-public true hzbtest+--------------------------------------+---------+-------------+-----------+| ID | Name | Description | Is_Public |+--------------------------------------+---------+-------------+-----------+| 6246eb36-4268-48d5-bbac-8da96cd3ea59 | hzbtest | - | True |+--------------------------------------+---------+-------------+-----------+
(nova-api)[root@cc07 /]# cinder extra-specs-list+--------------------------------------+----------+-------------------------------------+| ID | Name | extra_specs |+--------------------------------------+----------+-------------------------------------+| 0a220534-3404-40ff-b0a0-a449429e13c1 | ceph | { 'volume_backend_name': 'ceph'} || 1bcf273a-79de-4847-9dce-57628b2cd185 | vmdk | { 'volume_backend_name': 'vmdk'} || 6246eb36-4268-48d5-bbac-8da96cd3ea59 | hzbtest | {} || 747632fd-e8f2-403a-978d-892002498497 | ceph_new | { 'volume_backend_name': 'ceph_new'} || 88d1db93-1104-489c-ac9c-389a4e4736a1 | ceph-ssd | { 'volume_backend_name': 'ceph-ssd'} || 9b59e61e-b588-43fc-9c47-16337b595e74 | iscsi | { 'volume_backend_name': 'iscsi'} || b1c40afc-a8e3-4b78-b534-5d73a8ba98a8 | cephSsd | { 'volume_backend_name': 'cephSsd'} |+--------------------------------------+----------+-------------------------------------+
△type-key
新创建的volumeType并没有指定对接的是哪一个volume_backend。这时我们需要设置一个
usage: cinder type-key[ ...]Sets or unsets extra_spec for a volume type.Positional arguments: Name or ID of volume type. The action. Valid values are "set" or "unset." The extra specs key and value pair to set or unset. For unset, specify only the key.
(nova-api)[root@cc07 /]# cinder type-key hzbtest set volume_backend_name=ceph(nova-api)[root@cc07 /]# cinder extra-specs-list+--------------------------------------+----------+-------------------------------------+| ID | Name | extra_specs |+--------------------------------------+----------+-------------------------------------+| 0a220534-3404-40ff-b0a0-a449429e13c1 | ceph | { 'volume_backend_name': 'ceph'} || 1bcf273a-79de-4847-9dce-57628b2cd185 | vmdk | { 'volume_backend_name': 'vmdk'} || 6246eb36-4268-48d5-bbac-8da96cd3ea59 | hzbtest | {'volume_backend_name': 'ceph'} || 747632fd-e8f2-403a-978d-892002498497 | ceph_new | { 'volume_backend_name': 'ceph_new'} || 88d1db93-1104-489c-ac9c-389a4e4736a1 | ceph-ssd | { 'volume_backend_name': 'ceph-ssd'} || 9b59e61e-b588-43fc-9c47-16337b595e74 | iscsi | { 'volume_backend_name': 'iscsi'} || b1c40afc-a8e3-4b78-b534-5d73a8ba98a8 | cephSsd | { 'volume_backend_name': 'cephSsd'} |+--------------------------------------+----------+-------------------------------------+
取消设置
nova-api)[root@cc07 /]# cinder type-key hzbtest unset volume_backend_name=ceph
△删除volumeType
cinder type-delete 6246eb36-4268-48d5-bbac-8da96cd3ea59
Volume
△显示存储卷列表(和nova volume-list命令功能相同)
cinder list
△创建存储卷
cinder create --name VOLNAME SIZE(SIZE的单位为GB)
(nova-api)[root@cc07 /]# cinder create --name hzb-1g --volume-type ceph 1+------------------------------+--------------------------------------+| Property | Value |+------------------------------+--------------------------------------+| attachments | [] || availability_zone | nova || bootable | false || consistencygroup_id | None || created_at | 2018-10-29T02:14:28.000000 || description | None || encrypted | False || id | 48e23242-6ce6-486f-8841-417693b5f37c || metadata | {} || multiattach | False || name | hzb-1g || os-vol-tenant-attr:tenant_id | 26a8e1feb372493d831c19d4d9e28d73 || replication_status | disabled || size | 1 || snapshot_id | None || source_volid | None || status | creating || updated_at | None || user_id | d5fa609f7ffd4454b45b555940f41594 || volume_type | ceph |+------------------------------+--------------------------------------+(nova-api)[root@cc07 /]# cinder list+--------------------------------------+-----------+--------+------+-------------+----------+-------------+| ID | Status | Name | Size | Volume Type | Bootable | Attached to |+--------------------------------------+-----------+--------+------+-------------+----------+-------------+| 48e23242-6ce6-486f-8841-417693b5f37c | available | hzb-1g | 1 | ceph | false | |+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
△删除存储卷
cinder delete VOLNAME-OR-ID
cinder force-delete VOLNAME-OR-ID(强制删除)
△重命名存储卷
cinder rename VOLNAME-OR-ID NEW-VOLNAME
△显示存储卷信息
cinder show VOLNAME-OR-ID
△存储卷元数据
cinder metadata VOLNAME-OR-ID set KEY=VALUE
cinder metadata VOLNAME-OR-ID unset KEY
△创建存储卷备份
cinder backup-create --display-name BACKUP-VOLNAME VOLNAME-OR-ID
△删除存储卷备份
cinder backup-delete VOLNAME-OR-ID
△显示存储卷备份信息
cinder backup-show VOLNAME-OR-ID
△显示存储卷备份列表
cinder backup-list
△创建存储卷快照
cinder snapshot-create --display-name SNAPSHOT-VOLNAME VOLNAME-OR-ID
△删除存储卷快照
cinder snapshot-delete VOLNAME-OR-ID
△重命名存储卷快照
cinder snapshot-rename VOLNAME-OR-ID NEW-VOLNAME
△显示存储卷快照信息
cinder snapshot-show VOLNAME-OR-ID
△显示存储卷快照列表
cinder snapshot-list
△上传存储卷作为镜像
cinder upload-to-image VOLNAME-OR-ID IMAGE-NAME-OR-ID
△重置存储卷的status
假如有一个volume,状态 为in-use,但是确没有具体的虚拟机在使用,这是一种僵尸卷,无法删除,这时候需要重置状态为available才能删除
cinder reset-state ad7876d3-5a2a-46ee-8127-df8f60f0f571