系统总算是恢复了,但是在创建实例的时候是有一个节点创建成功,其他节点报错如下。
节点的报错日志
[root@compute-1 nova]# cat nova-compute.log
2020-12-18 19:18:25.036 3284 ERROR nova.scheduler.client.report [req-5ac80bc6-08ae-4041-ba63-7bf312ce8404 - - - - -] [req-d780e2eb-6ebf-49b6-be40-5e6693379227] Failed to create resource provider record in placement API for UUID dd683ade-dbfc-4dff-b227-a18c811405d7. Got 409: {"errors": [{"status": 409, "request_id": "req-d780e2eb-6ebf-49b6-be40-5e6693379227", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: compute-1 already exists. ", "title": "Conflict"}]}.
2020-12-18 19:18:25.037 3284 ERROR nova.compute.manager [req-5ac80bc6-08ae-4041-ba63-7bf312ce8404 - - - - -] Error updating resources for node compute-1.rocky2.yfb0801800.islab.net.cn.: ResourceProviderCreationFailed: Failed to create resource provider compute-1
[root@controller nova]# cat nova-scheduler.log
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: ServiceNotFound: Service 4 could not be found.
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db service.service_ref.save()
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 397, in save
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 183, in service_update
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db self.force_reraise()
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 228, in wrapped
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 557, in service_update
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 243, in wrapped
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 420, in service_get
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db raise exception.ServiceNotFound(service_id=service_id)
2020-12-18 16:41:09.544 3216 ERROR nova.servicegroup.drivers.db ServiceNotFound: Service 4 could not be found.
以上日志提示4台compute几点没有发现。
查阅了部分资料发现是resource_providers的uuid 和compute ID不一致所造成的。究其原因可能有两点,一是我在重新刷数据库是没有停止相关的服务,然后在刷新数据库后之前的缓存数据直接被写入的数据库;二是在数据库重建后没有及时的重启计算节点的服务,可到会导致把原来的resource_providers ID提前刷入数据库。
相关参考资料
https://www.cnblogs.com/yanling-coder/p/11908573.html (中文)
https://docs.openstack.org/osc-placement/latest/cli/index.html (官方)
修复过程
#yum install -y python2-osc-placement.noarch (注意你是python3还是python2)
或者
#pip install osc-placement
这个定义在部分版本中可以不用理睬。
$ export OS_PLACEMENT_API_VERSION=1.10
$ openstack allocation candidate list --resource VCPU=1
#查看 service ID
[root@controller ~]# nova service-list
+--------------------------------------+------------------+-------------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+-------------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 118a7f0a-5815-427f-9c2a-1ae335773a9e | nova-consoleauth | controller | internal | enabled | up | 2020-12-21T04:15:53.000000 | - | False |
| fb41d2c7-47f1-49f3-ae35-53008b019947 | nova-scheduler | controller| internal | enabled | up | 2020-12-21T04:15:54.000000 | - | False |
| d930595e-4886-422b-b33d-1e75c16d3d61 | nova-conductor | controller| internal | enabled | up | 2020-12-21T04:15:57.000000 | - | False |
| e80a992d-42c8-48fa-b467-76db2b74d3f2 | nova-compute | compute-4 | nova | enabled | up | 2020-12-21T04:15:52.000000 | - | False |
| 9e76c396-250c-4dd0-9d88-1f27bdb0a5a5 | nova-compute | compute-5| nova | enabled | up | 2020-12-21T04:15:57.000000 | - | False |
| 36d5fb08-707e-4f68-ab71-fd6c895ecf08 | nova-compute | compute-3 | nova | enabled | up | 2020-12-21T04:15:53.000000 | - | False |
| f9fc99b0-f8b8-4e7e-85c0-85cbcf77e53e | nova-compute | compute-2 | nova | enabled | up | 2020-12-21T04:15:52.000000 | - | False |
| 3c0922ea-253f-41e4-98b0-6934df15f2f5 | nova-compute | compute-1 | nova | enabled | up | 2020-12-21T04:15:52.000000 | - | False |
+--------------------------------------+------------------+-------------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
#查看 resource provider ID (compute已经被修复了)
[root@controller ~]# openstack resource provider list
+--------------------------------------+------------------------------------------+------------+
| uuid | name | generation |
+--------------------------------------+------------------------------------------+------------+
| 9b6bf51c-d443-459a-9d86-c2ed0d1d5793 | compute-4 | 2 |
| 16afc275-e404-406e-aafb-4e1c5f30763f | compute-5 | 2 |
| b1c484e8-58cb-41d5-84d3-1e1f4ffab341 | compute-3 | 2 |
| cb90f3c2-9ed4-4a1a-af2f-44ed930958a7 | compute-2 | 10 |
| 13228c64-2ccd-439a-ae55-0e927e4e64b9 | compute-1 | 6 |
+--------------------------------------+------------------------------------------+------------+
##首先先停止compute node openstack service.
[root@compute-3 nova]# systemctl stop libvirtd.service openstack-nova-compute.service
##删除 service ID
##nova service-delete <service ID>
[root@controller ~]# nova service-delete 36d5fb08-707e-4f68-ab71-fd6c895ecf
##删除 resource provider UUID
##openstack resource provider delete <UUID>
[root@controller ~]# openstack resource provider delete b1c484e8-58cb-41d5-84d3-1e1f4ffab341
重启compute nova服务后可恢复正常
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)