您可以使用managed_mapped_file
从内存映射文件透明地分配。
这意味着,出于所有实际目的,您通常不需要重新划分内存区域。无论如何,这都是虚拟内存,因此分页负责在需要的时间加载正确的位。
显然,如果存在大量碎片或访问“跳跃”,那么分页可能会成为性能瓶颈。在这种情况下,请考虑细分为池并从中分配。)_
Edit刚刚注意到 Boost IPC 在下面对此有支持隔离存储节点分配器 http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.stl_allocators_segregated_storage and 自适应池节点分配器 http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.stl_allocators_adaptive。还有关于这些存储池的实现的注释here http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/architecture.html#interprocess.architecture.allocators_containers.implementation_segregated_storage_pools.
这是一个简单的起点,创建一个 50Gb 文件并在其中填充一些数据:
#include <iostream>
#include <string>
#include <vector>
#include <iterator>
#include <algorithm>
#include <boost/container/flat_map.hpp>
#include <boost/container/flat_set.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/sync/named_mutex.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
namespace bip = boost::interprocess;
using mutex_type = bip::named_mutex;
struct X
{
char buf[100];
double rate;
uint32_t samples[1024];
};
template <typename T> using shared_alloc = bip::allocator<T,bip::managed_mapped_file::segment_manager>;
template <typename T> using shared_vector = boost::container::vector<T, shared_alloc<T> >;
template <typename K, typename V, typename P = std::pair<K,V>, typename Cmp = std::less<K> >
using shared_map = boost::container::flat_map<K, V, Cmp, shared_alloc<P> >;
using shared_string = bip::basic_string<char,std::char_traits<char>,shared_alloc<char> >;
using dataset_t = shared_map<shared_string, shared_vector<X> >;
struct mutex_remove
{
mutex_remove() { mutex_type::remove("7FD6D7E8-320B-11DC-82CF-39598D556B0E"); }
~mutex_remove(){ mutex_type::remove("7FD6D7E8-320B-11DC-82CF-39598D556B0E"); }
} remover;
static mutex_type mutex(bip::open_or_create,"7FD6D7E8-320B-11DC-82CF-39598D556B0E");
static dataset_t& shared_instance()
{
bip::scoped_lock<mutex_type> lock(mutex);
static bip::managed_mapped_file seg(bip::open_or_create,"./demo.db", 50ul<<30); // "50Gb ought to be enough for anyone"
static dataset_t* _instance = seg.find_or_construct<dataset_t>
("DATA")
(
std::less<shared_string>(),
dataset_t::allocator_type(seg.get_segment_manager())
);
static auto capacity = seg.get_free_memory();
std::cerr << "Free space: " << (capacity>>30) << "g\n";
return *_instance;
}
int main()
{
auto& db = shared_instance();
bip::scoped_lock<mutex_type> lock(mutex);
auto alloc = db.get_allocator().get_segment_manager();
std::cout << db.size() << '\n';
for (int i = 0; i < 1000; ++i)
{
std::string key_ = "item" + std::to_string(i);
shared_string key(alloc);
key.assign(key_.begin(), key_.end());
auto value = shared_vector<X>(alloc);
value.resize(size_t(rand()%(1ul<<9)));
auto entry = std::make_pair(key, value);
db.insert(std::make_pair(key, value));
}
}
注意它写了一个sparse文件50G。提交的实际大小取决于那里的一些随机性。我的运行结果大约是 1.1G:
$ du -shc --apparent-size demo.db
50G demo.db
$ du -shc demo.db
1,1G demo.db
希望这可以帮助