将 HuggingFace Trainer 与 Ray Tune 结合使用时，“无法 pickle _thread.RLock 对象”

我尝试使用TrainerHuggingFace 变压器中的函数来训练模型并使用ray tune用于超参数搜索。我可以在 Google Colab 中运行代码而不会出现错误，但是当我使用远程计算机通过 ssh 运行代码时，会出现错误：

Traceback (most recent call last):
  File "main.py", line 130, in <module>
    backend="ray")
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/transformers/trainer.py", line 1668, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/transformers/integrations.py", line 236, in run_hp_search_ray
    **kwargs,
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/tune/tune.py", line 297, in run
    _ray_auto_init()
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/tune/tune.py", line 670, in _ray_auto_init
    ray.init()
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/worker.py", line 940, in init
    hook()
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/tune/registry.py", line 197, in flush
    self.references[k] = ray.put(v)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/worker.py", line 1597, in put
    object_ref = worker.put_object(value)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/worker.py", line 287, in put_object
    serialized_value = self.get_serialization_context().serialize(value)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 331, in serialize
    return self._serialize_to_msgpack(value)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 311, in _serialize_to_msgpack
    self._serialize_to_pickle5(metadata, python_objects)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 271, in _serialize_to_pickle5
    raise e
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/serialization.py", line 268, in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/cs/research/external/home/zhengzho/.local/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
TypeError: can't pickle _thread.RLock objects

多谢！

这是因为内存跟踪器Trainer用途不可腌制。

在最近的transformer版本（我相信是 4.6 及以上），默认情况下禁用内存跟踪器。

或者您可以通过设置手动禁用它skip_memory_metrics=True在你的TrainingArguments.

See https://github.com/huggingface/transformers/issues/12177 https://github.com/huggingface/transformers/issues/12177