ClickHouse介绍
-
ClickHouse
是由俄罗斯的Yandex
公司开发的一款快速、可扩展的列式数据库管理系统。它专门针对OLAP
场景设计,在海量数据分析和查询方面具有出色的性能表现,同时还能支持实时数据插入和更新。
-
ClickHouse
采用了面向列的存储方式,具有较高的压缩比和查询速度。它支持SQL
语言,并提供了一系列丰富的函数库,可满足各种复杂的数据分析需求。同时,ClickHouse
还支持数据分区和分片,可实现横向扩展
-
ClickHouse
可在Linux
和Windows
等操作系统上运行。它还支持RESTful API
和JDBC
等多种接口,方便与其他系统进行集成。由于其出色的性能和可扩展性,ClickHouse被许多企业用于大规模数据处理和分析,如Yandex
、CloudFlare
、Kenshoo
等。在国内,ClickHouse
也被各大厂青睐和使用,例如京东、腾讯、哔哩哔哩、字节跳动等
- 从我个人两年多的使用经验来说,它对于中小型公司,只有部分业务涉及大数据处理的场景,也很适用,我们可以使用单体
ClickHouse
作为大数据存储和分析组件,后续可以根据业务增长自由扩展
- 在使用
ClickHouse
时,我们可以尽情的存储我们的海量数据,也可以毫不忌讳的直接把海量数据使用SQL
进行计算和统计
- 有人这么形容clickhouse,说它是一个手动拼装的跑车,性能强劲但需要手动去维修维护它
- 目前遇到比较头疼的一个问题,就是断电重启导致的文件损坏问题,虽然在使用其他数据库时也遇到过,但是感觉异常断电时,它比MySQL等更容易出问题,毕竟压缩存储要求它一直在计算和合并文件
问题介绍
- 办公室有一台二手服务器,作为平时开发测试使用。由于机器没放在机房,会偶发断电
- 异常断电后,文件系统是有出问题的可能的,尤其是一些不断在读写合并的文件
- 春节后,发现clickhouse启动不了,使用
systemctl status clickhouse-server
返回如下
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
Loaded: loaded (/etc/systemd/system/clickhouse-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since 二 2023-02-21 15:46:41 CST; 10s ago
Process: 9688 ExecStart=/usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid (code=exited, status=70)
Main PID: 9688 (code=exited, status=70)
2月 21 15:46:41 localhost.localdomain systemd[1]: clickhouse-server.service: main process exited, code=exited, status=70/n/a
2月 21 15:46:41 localhost.localdomain systemd[1]: Unit clickhouse-server.service entered failed state.
2月 21 15:46:41 localhost.localdomain systemd[1]: clickhouse-server.service failed.
- 对于具体的报错信息,还是要查看clickhouse的报错日志,默认位置在
/var/log/clickhouse-server/
,错误日志如下:
2023.02.21 15:47:12.351823 [ 9848 ] {} <Error> system.query_thread_log (95febdac-d99a-4bdb-9431-653a75f3a34b): Detaching broken part /var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394084_2980 (size: 0.00 B). If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2023.02.21 15:47:12.365662 [ 9787 ] {} <Error> Application: Caught exception while loading metadata: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in directory_iterator::directory_iterator(...): Structure needs cleaning [/var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394023_2919/], Stack trace (when copying this message, always include the lines below):
0. std::runtime_error::runtime_error(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1b4107cd in ?
1. std::__1::system_error::system_error(std::__1::error_code, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1b41a437 in ?
2. ? @ 0x145458c1 in /usr/bin/clickhouse
3. ? @ 0x1b3bf356 in /usr/bin/clickhouse
4. ? @ 0x1b3bd762 in /usr/bin/clickhouse
5. std::__1::__fs::filesystem::directory_iterator::directory_iterator(std::__1::__fs::filesystem::path const&, std::__1::error_code*, std::__1::__fs::filesystem::directory_options) @ 0x1b3bd581 in /usr/bin/clickhouse
6. DB::DiskLocalDirectoryIterator::DiskLocalDirectoryIterator(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x14552124 in /usr/bin/clickhouse
7. DB::DiskLocal::iterateDirectory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x145483d7 in /usr/bin/clickhouse
8. DB::MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(std::__1::shared_ptr<DB::IDisk> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1552dd06 in /usr/bin/clickhouse
9. DB::MergeTreeData::createPart(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::MergeTreePartInfo const&, std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::IMergeTreeDataPart const*) const @ 0x1547a871 in /usr/bin/clickhouse
10. ? @ 0x154c98ac in /usr/bin/clickhouse
11. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xaf6546a in /usr/bin/clickhouse
12. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&&...)::'lambda'()::operator()() @ 0xaf674a4 in /usr/bin/clickhouse
13. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xaf62837 in /usr/bin/clickhouse
14. ? @ 0xaf662fd in /usr/bin/clickhouse
15. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so
16. __clone @ 0xfe96d in /usr/lib64/libc-2.17.so
Cannot print extra info for Poco::Exception (version 22.2.2.1)
2023.02.21 15:47:12.366162 [ 9787 ] {} <Information> Application: Shutting down storages.
2023.02.21 15:47:12.366178 [ 9787 ] {} <Information> Context: Shutdown disk default
2023.02.21 15:47:12.391451 [ 9787 ] {} <Debug> Application: Shut down storages.
2023.02.21 15:47:13.199583 [ 9787 ] {} <Debug> Application: Destroyed global context.
2023.02.21 15:47:13.202239 [ 9787 ] {} <Error> Application: filesystem error: in directory_iterator::directory_iterator(...): Structure needs cleaning [/var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394023_2919/]
2023.02.21 15:47:13.202291 [ 9787 ] {} <Information> Application: shutting down
2023.02.21 15:47:13.202308 [ 9787 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2023.02.21 15:47:13.203474 [ 9788 ] {} <Trace> BaseDaemon: Received signal -2
2023.02.21 15:47:13.203546 [ 9788 ] {} <Information> BaseDaemon: Stop SignalListener thread
2023.02.21 15:47:13.276203 [ 9786 ] {} <Information> Application: Child process exited normally with code 70.
- 关键报错日志是这一句:
<Error> Application: filesystem error: in directory_iterator::directory_iterator(...): Structure needs cleaning [/var/lib/clickhouse/store/95f/95febdac-d99a-4bdb-9431-653a75f3a34b/202302_390451_394023_2919/]
,这个文件夹损坏,结构需要清理
解决办法
- 百度了一堆,确实能搜到解决方式,牵涉到磁盘挂载、修复之类的,试了下没用,识别不到磁盘,可能是我哪里搞错了,暂时舍弃
- 还有一个办法,给文件重命名,试了下,使用
mv old_file_name new_file_name
重命名,直接死机了
- 重启服务器后,我使用文件传输工具查看修改文件名,竟然成功了。想了下这个损坏的文件,应该不能随便移动,应该使用
rename
- 虽然文件命名成功了,但是clickhouse还是无法启动,还是类似报错,应该是clickhouse还是会使用这个文件
- 想着怎么才能让clickhouse忽略这个文件夹,试着把上面层级的文件夹
95f
改了下,改成95f1
,启动服务后发现可以了