程序员经验分享-hwhale

如何在 Amazon EMR 上安装多个版本的 numpy 以及如何删除早期版本？

2023-12-07

我不明白 Python 如何安装单个包的多个版本，或者为什么当我安装了多个版本时，import package没有给我最新的。

我正在使用 AWS linux 和 AWS EMR 中的 AWS 存储库。当我安装 Python 3.6 时，它默认提供 Numpy 版本 1.14.5。我想使用版本 1.19.1 并将其放在一个目录中。

$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/packages/ numpy==1.19.1
Requirement already up-to-date: numpy==1.19.1 in /usr/local/lib64/python3.6/site-packages
$

但请注意：

 $ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.14.5'
>>>

因为我有both安装了版本 1.14.5 和版本 1.19.1。

我可以卸载版本 1.19.1，然后可以卸载版本 1.14.5，尽管看起来我必须卸载它两次（请参阅下面的扩展混音）。

那么我该如何做到这一点import numpy获取版本1.19.1而不是旧版本？

$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib64/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib64/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/__version__.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/compat.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/environment.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/f2py/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/fft/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/tests/test_fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/lib/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/linalg/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/ma/version.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/randomkit.h
  /usr/local/lib64/python3.6/site-packages/numpy/testing/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/__init__.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/nosetester.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/parameterized.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/utils.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nosetester.py
Proceed (y/n)? y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib/python3.6/site-packages/numpy/LICENSE.txt
  /usr/local/lib/python3.6/site-packages/numpy/__config__.py
  /usr/local/lib/python3.6/site-packages/numpy/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/_distributor_init.py
  /usr/local/lib/python3.6/site-packages/numpy/_globals.py
  /usr/local/lib/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/_inspect.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/py3k.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/conftest.py
  /usr/local/lib/python3.6/site-packages/numpy/core/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib/python3.6/site-packages/numpy/core/_internal.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_methods.py
  /usr/local/lib/python3.6/site-packages/numpy/core/arrayprint.py
  /usr/local/lib/python3.6/site-packages/numpy/core/cversions.py
  /usr/local/lib/python3.6/site-packages/numpy/core/defchararray.py
  /usr/local/lib/python3.6/site-packages/numpy/core/einsumfunc.py
  /usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py
  /usr/local/lib/python3.6/site-packages/numpy/core/function_base.py
  /usr/local/lib/python3.6/site-packages/numpy/core/generate_numpy_api.py
  /usr/local/lib/python3.6/site-packages/numpy/core/getlimits.py
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__multiarray_api.h
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__ufunc_api.h
...
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/lahey.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/mips.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/nag.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/none.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pathf95.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pg.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/sun.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/vast.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/from_template.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/intelccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/lib2def.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/line_endings.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/log.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw/gfortran_vs2003_hack.c
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw32ccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvc9compiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/numpy_distribution.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/pathccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_exec_command.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_gnu.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_nagfor.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/unixccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/basics.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/broadcasting.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/byteswapping.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/constants.py
...
  /usr/local/lib/python3.6/site-packages/numpy/testing/print_coercion_tables.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_decorators.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_doctesting.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_utils.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/utils.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_ctypeslib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_matlib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_numpy_version.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_reloading.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_scripts.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_warnings.py
  /usr/local/lib/python3.6/site-packages/numpy/version.py
Proceed (y/n)? Y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/gits/das-vm-config/dist/python numpy==1.19.1
Collecting numpy==1.19.1
Installing collected packages: numpy
Successfully installed numpy-1.19.1
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.19.1'
>>>

我们的解决方案是在 EMR 引导脚本（在每个节点上运行）中执行以下操作：

  1. 使用yum安装python36和python36-pip
  2. 使用 pip-3.6 与--target安装选项numpy-1.19.1 in /mnt/python_additions。我们在引导操作中执行此操作，以便它发生在每个节点上。
  3. 让亚马逊的脚本安装python36-numpy，安装numpy-1.14.4在站点库位置。
  4. 在被调用的程序中spark-submit, add /mnt/python_additions to sys.path.

这是一个 hack，但它确实有效。

本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)

python

python3x

NumPy

pip

如何在 Amazon EMR 上安装多个版本的 numpy 以及如何删除早期版本？ 的相关文章

随机推荐

  • 读取系统调用表函数地址时内核模块崩溃

    我正在研究 rootkit 并尝试挂钩系统调用表 由于我已经可以从 boot System map uname r 动态检索表的地址 因此我跟踪了代码的有问题的部分并将其隔离到一个独立的 更简单的模块中 如下所示 它尝试检索并显示 Kill

  • 如何计算R中特定行的平均值？

    我有一个数据文件 如下例所示 但更大 names num Y1 Y2 William 1 4 71 7 4 William 2 3 75 8 William 3 4 71 7 9 Katja 1 5 83 8 5 Katja 2 5 17

  • VStack 中大 Text 和 TextField 之间的 SwiftUI 神秘间距

    我无法弄清楚为什么我的文本下方有一些空格 struct testView View State private var notes var body some View VStack Text Larg Text font system s

  • 为什么这个值是空的？

    我正在成功地制作 保存和检索我的共享偏好mainActivity 但我无法从我的服务中获取它 由于某种原因 当我尝试从后台服务检索它时 我的共享首选项为空 我在 onCreate 中初始化我的首选项 contactsPrefs getSha

  • AngularJS：链接承诺

    遵循以下建议AngularJS 验证和承诺 我想链接确认对话框 从而一次验证多个步骤 根据用户提供的数据 调用API来查看哪些内容需要用户确认 对于每一个需要确认的步骤 提示用户并让他们决定是否进入下一步 如果任何步骤返回 false 则整

  • 如何替换 perl 中的空格

    chomp myString myString s g 我可以用这两个替换吗 myString s s g 有什么区别吗 请解释 您的第一个代码将从 myString 末尾取出换行符 如果存在 然后删除所有 字符 第二行代码将删除所有空白字

  • 为什么整数 0 等于 PHP 中的字符串？ [复制]

    这个问题在这里已经有答案了 可能的重复 相等 双等于 和恒等 三等于 比较运算符有何不同 Why this var dump 0 string 输出这个 bool true 上下文不是 运算符应该转换0 into FALSE and str

  • 删除重写方法中的抛出，编译器在调用时需要一个 try/catch 块

    我有一个子类 并且要重写继承的父方法 我要从方法声明中删除 throws 子句 现在 使用多态性 my 实例的运行时类型应该确定方法实现 然而 当我尝试编译时 编译器抱怨并希望在方法调用周围有一个 try catch 块 就好像正在调用超类

  • 尝试从 PHP 运行 jar

    在阅读了该网站上与我遇到的完全相同的问题相关的一些帖子后 我发现它们都没有给我一个成功的结果 如何使用 PHP 从 Web 服务器运行 jar 从网站上的 PHP 脚本运行 Java 类文件 为什么 exec java jar file j

  • 制作特定结构的矩阵

    请注意 我不知道我错在哪里 但我花了一整天的时间试图解决这个问题 因此 我请求不要将其作为重复问题丢弃 并将其视为与矩阵结构相关的非常具体的问题 我有以下数据框 dput c m q structure list ASK Price c 1

  • 为什么 JCheckBox 上的 setSelected 失去作用？

    有人可以向我解释为什么我失去了选择 由setSelected for JCheckBox当我把JOptionPane进入ItemListener 这是一个错误吗 奇怪的是 如果这个过程被延迟invokeLater setSelected 正

  • Firemonkey：如何定义一个包含另一个组件的组件？

    在 Delphi 下 我想创建一个新的 firemonkey 控件 其中将包含另一个 firemonkey 控件 这并不是真正的问题 因为我可以这样做 constructor TMyComponent Create AOwner TComp

  • 更新对象图时实体框架的断开行为

    我目前正在开发一个使用以下技术的项目 ASP net MVC 表示层 数据服务层 WCF 具有 Automapper 的数据传输对象 TO 层 领域层 POCO 代码优先实体框架 存储库层 实体框架4 3 DbContext 我们使用 DT

  • 使用 EF Core 和 NpgSql 过滤 postgres 中 jsonb 列的整数数组

    我想根据 json 整数数组过滤行 例如 我的表如下所示 Id Name TypeJson 1 Name One 1 2 2 Name Two 2 3 3 Name Three 4 7 其中 Id 是 int 类型 Name 文本 Type

  • 在 Octave 中导入 Java 类

    我一直很难弄清楚如何做到这一点 从八度网站 似乎java类是通过类路径找到的 这个堆栈溢出答案表示 静态java路径 是 动态java路径 但我不确定如何设置静态 java 路径 在我感兴趣的特定情况下 我尝试将 javaplex 包与 O

  • 当pdf加载到iframe时如何阻止下载？

    当iframe加载pdf时自动下载pdf文件 我怎样才能防止这种情况 function dialog dialog autoOpen false modal true resizable false width auto show fade

  • 我的输出没有显示完整的数字，但显示 ??反而

    I m trying to do benchmarking with JMH the benchmarking result did come out but not perfectly 不知何故有 在数字中 是否是我的 IDE 的问题 我

  • 大熊猫的大小和数量有什么区别？

    这就是之间的区别groupby x count and groupby x size在熊猫中 size 是否只排除 nil size包括NaN价值观 count才不是 In 46 df pd DataFrame a 0 0 1 2 2 2

  • 过滤 ElementsCollection

    我正在尝试创建一个函数来过滤ElementsCollection 条件是每个元素的子元素而不是元素本身 这是我想出的 public static ElementsCollection filterByChild ElementsCollec

  • 如何在 Amazon EMR 上安装多个版本的 numpy 以及如何删除早期版本？

    我不明白 Python 如何安装单个包的多个版本 或者为什么当我安装了多个版本时 import package没有给我最新的 我正在使用 AWS linux 和 AWS EMR 中的 AWS 存储库 当我安装 Python 3 6 时 它默

热门标签