Navigation

    Gpushare.com

    • Register
    • Login
    • Search
    • Popular
    • Categories
    • Recent
    • Tags

    多线程解压缩 pigz

    技术交流
    2
    2
    57
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • 188****7632
      188****7632 last edited by

      pigz支持多线程并行处理
      之前处理好的数据大约50G左右,记得用zip压缩差不多要1个小时,用pigz不到10分钟,8w条数据,大文件压缩速度效果挺明显的

      sudo apt-get update -y
      sudo apt-get install -y pigz
      


      用coco数据集测试一下

      经过测试,
      pigz 大约1分钟压缩 1.3g左右
      zip 大约1分钟压缩 1.1g左右
      因为colab这个cpu只有2个线程,而推荐的是8个线程起步,所以速度差距没有那么明显😧

      参考:
      https://wangchujiang.com/linux-command/c/pigz.html
      https://zoomadmin.com/HowToInstall/UbuntuPackage/pigz

      1 Reply Last reply Reply Quote 3
      • Alice_恒源云
        Alice_恒源云 last edited by

        这个方法挺实用的~

        1 Reply Last reply Reply Quote 0
        • First post
          Last post