Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.2k views
in Technique[技术] by (71.8m points)

bash - Install everything from a `conda list` output

I have a file, conda_env.txt, resulting from saving the stdout of conda list. It has the following structure.

conda_env.txt

# packages in environment at /scratch/hello/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0
absl-py                   0.7.1                    pypi_0    pypi
alabaster                 0.7.12                   py37_0
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.7                    py37_0
anaconda-project          0.8.2                    py37_0
annoy                     1.14.0                   pypi_0    pypi

I would like to conda install all the packages listed in this file, with version specified.

My idea is to for loop over all lines (starting from the 4th line), get the value from the first and second field, and save them into PACKAGE and VERSION, and install the package by calling conda install $PACKAGE=$VERSION (for example, conda install pandas=0.20.3

I would like to write a shell script to do this. I have written a demo file just to read the ith line and it's second field:

#!/bin/bash
LINES=$(wc -l < conda_env.txt)
for i in {4...10}
do
awk -F '|' 'NR==${i}{print $2}' conda_env.txt
done

and it gives me error: awk: bailing out at source line 1.

Any idea what is going wrong here and how to do it correctly?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Convert to YAML

I wrote an AWK script to convert the output of conda list to YAML that is compatible with conda env create -f env.yaml. This should be strongly preferred over a sequence of conda install calls, which results in running a solve for each individual package.

AWK Script

list_to_yaml.awk

#!/usr/bin/env awk -f
#' Author: Mervin Fansler
#' GitHub: @mfansler
#' License: MIT

#' Usage
#' $ conda list | awk -f list_to_yaml.awk

{
  # skip header
  if ($1 ~ /^#/) { next }

  if ($4 ~ /pypi/) {  # pypi packages
    pip=1;
    pypi[i++]="    - "$1"=="$2" ";
  } else {  # conda packages
    if ($1 ~ /pip/) {
      pip=1;
    } else {
      conda[j++]="  - "$1"="$2" ";
    }
    
    # include channels
    if (!seen[$4]) {
      if (length($4) == 0) {
        channels[k++]="  - defaults ";
      } else {
        channels[k++]="  - "$4" ";
      }
      ++seen[$4];
    }
  }
}
END {
  # emit channel info
  print "channels: ";
  for (k in channels) print channels[k];
  
  # emit conda pkg info                                                                                                               
  print "dependencies: ";
  for (j in conda) print conda[j];

  # emit PyPI pkg info
  if (pip) print "  - pip ";
  if (length(pypi) > 0) {
    print "  - pip: ";
    for (i in pypi) print pypi[i];
  }
}

Usage

Suppose we have an environment foo that someone dumped a conda list for (foo_list.txt), we can create a corresponding YAML file (foo.yaml), and then use that to generate a new environment, bar.

# dump a `conda list` for environment `foo`
conda list -n foo > foo_list.txt

# convert to YAML
awk -f list_to_yaml.awk foo_list.txt > foo.yaml

# edit the foo.yaml channels section manually!
# usually should be `conda-forge` at the top, `defaults` at the bottom
# emacs foo.yaml

# create the new environment
conda env create -n bar -f foo.yaml

Alternatively, if you just want to test it, piping also works:

conda list -n foo | awk -f list_to_yaml.awk

Advantages

  • automatically handles packages from PyPI
  • handles channels (sort of; the order may not be optimal)
  • creates a single file for installation
  • doesn't repeatedly call conda install, which leads to a ton of unnecessary solves

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...