我有一个从文件中顺序读取数据的应用程序。有些是直接从指向的指针读取mmap
ed 文件和其他部分是memcpy
ed 从文件到另一个缓冲区。我注意到在进行大型操作时性能不佳memcpy
我需要的所有内存(1MB 块)以及在执行大量较小操作时的更好性能memcpy
调用(在我的测试中,我使用了 4KB,页面大小,这花费了 1/3 的时间来运行。)我相信问题是使用大的页面时会出现大量的主要页面错误memcpy
.
我尝试过各种调整参数(MAP_POPUATE
, MADV_WILLNEED
, MADV_SEQUENTIAL
)没有任何明显的改善。
我不知道为什么很多小memcpy
通话速度应该更快;这似乎违反直觉。有什么办法可以改善这一点吗?
结果和测试代码如下。
在 CentOS 7 (linux 3.10.0) 上运行,默认编译器 (gcc 4.8.5),从常规磁盘的 RAID 阵列中读取 29GB 文件。
跑步与/usr/bin/time -v
:
4KB memcpy
:
User time (seconds): 5.43
System time (seconds): 10.18
Percent of CPU this job got: 75%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:20.59
Major (requiring I/O) page faults: 4607
Minor (reclaiming a frame) page faults: 7603470
Voluntary context switches: 61840
Involuntary context switches: 59
1MB memcpy
:
User time (seconds): 6.75
System time (seconds): 8.39
Percent of CPU this job got: 23%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:03.71
Major (requiring I/O) page faults: 302965
Minor (reclaiming a frame) page faults: 7305366
Voluntary context switches: 302975
Involuntary context switches: 96
MADV_WILLNEED
似乎对1MB复制结果没有太大影响。
MADV_SEQUENTIAL
1MB 的复制结果减慢了很多,我没有等到它完成(至少 7 分钟)。
MAP_POPULATE
将 1MB 复制结果减慢了约 15 秒。
用于测试的简化代码:
#include <algorithm>
#include <iostream>
#include <stdexcept>
#include <fcntl.h>
#include <stdint.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
int
main(int argc, char *argv[])
{
try {
char *filename = argv[1];
int fd = open(filename, O_RDONLY);
if (fd == -1) {
throw std::runtime_error("Failed open()");
}
off_t file_length = lseek(fd, 0, SEEK_END);
if (file_length == (off_t)-1) {
throw std::runtime_error("Failed lseek()");
}
int mmap_flags = MAP_PRIVATE;
#ifdef WITH_MAP_POPULATE
mmap_flags |= MAP_POPULATE; // Small performance degredation if enabled
#endif
void *map = mmap(NULL, file_length, PROT_READ, mmap_flags, fd, 0);
if (map == MAP_FAILED) {
throw std::runtime_error("Failed mmap()");
}
#ifdef WITH_MADV_WILLNEED
madvise(map, file_length, MADV_WILLNEED); // No difference in performance if enabled
#endif
#ifdef WITH_MADV_SEQUENTIAL
madvise(map, file_length, MADV_SEQUENTIAL); // Massive performance degredation if enabled
#endif
const uint8_t *file_map_i = static_cast<const uint8_t *>(map);
const uint8_t *file_map_end = file_map_i + file_length;
size_t memcpy_size = MEMCPY_SIZE;
uint8_t *buffer = new uint8_t[memcpy_size];
while (file_map_i != file_map_end) {
size_t this_memcpy_size = std::min(memcpy_size, static_cast<std::size_t>(file_map_end - file_map_i));
memcpy(buffer, file_map_i, this_memcpy_size);
file_map_i += this_memcpy_size;
}
}
catch (const std::exception &e) {
std::cerr << "Caught exception: " << e.what() << std::endl;
}
return 0;
}