Don’t Let Conda Eat Your Hard Drive

If you happen to’re an Anaconda consumer, that conda environments allow you to handle bundle dependencies, keep away from compatibility conflicts, and share your tasks with others. Sadly, they will additionally take over your pc’s arduous drive.

I write a number of pc tutorials and to maintain them organized, every has a devoted folder construction full with a Conda Environment. This labored nice at first, however quickly my pc’s efficiency degraded, and I seen that my SSD was filling up. At one level I had solely 13 GB free.

Conda helps handle this drawback by storing downloaded bundle information in a single “cache” (pkgs_dirs). While you set up a bundle, conda checks for it within the bundle cache earlier than downloading. If not discovered, conda will obtain and extract the bundle and hyperlink the information to the lively atmosphere. As a result of the cache is “shared,” completely different environments can use the identical downloaded information with out duplication.

As a result of conda caches each downloaded bundle, pkgs_dirs can develop to many gigabytes. And whereas conda hyperlinks to shared packages within the cache, there’s nonetheless a must retailer some packages within the atmosphere folder. That is primarily to keep away from model conflicts, the place completely different environments want completely different variations of the identical dependency (a bundle required to run one other bundle).

As well as, massive, compiled binaries like OpenCV could require full copies within the atmosphere’s listing, and every atmosphere requires a replica of the Python interpreter (at 100–200 MB). All these points can bloat conda environments to a number of gigabytes.

On this Fast Success Information Science venture, we’ll have a look at some methods for decreasing the storage necessities for conda environments, together with these saved in default areas and devoted folders.

---

Reminiscence Administration Methods

Beneath are some Memory Management methods that can allow you to scale back conda’s storage footprint in your machine. We’ll talk about every in flip.

1. Cache cleansing
2. Sharing task-based environments
3. Archiving with atmosphere and specs information
4. Archiving environments with conda-pack
5. Storing environments on an exterior drive
6. Relocating the bundle cache
7. Utilizing digital environments (venv)

1. Cleansing the Bundle Cache

Cleansing the bundle cache is the primary and best step for liberating up reminiscence. Even after deleting environments, conda retains the associated bundle information within the cache. You may release area by eradicating these unused packages and their related tarballs (compressed bundle information), logs, index caches (metadata saved in conda), and momentary information.

Conda permits an non-obligatory “dry run” to see how a lot reminiscence will likely be reclaimed. You’ll wish to run this from both the terminal or Anaconda Immediate in your base atmosphere:

conda clear --all --dry-run

To commit, run:

conda clear --all

Right here’s how this appears on my machine:

This course of trimmed a wholesome 6.28 GB and took a number of minutes to run.

2. Sharing Job-based Environments

Creating a couple of environments for specialised duties — like pc imaginative and prescient or geospatial work — is extra reminiscence environment friendly than utilizing devoted environments for every venture. These environments would come with primary packages plus ones for the precise job (reminiscent of OpenCV, scikit-image, and PIL for pc imaginative and prescient).

A bonus of this strategy is that you may simply maintain all of the packages updated and hyperlink the environments to a number of tasks. Nonetheless, this received’t work if some tasks require completely different variations of the shared packages.

3. Archiving with Surroundings and Specs Information

If you happen to don’t have sufficient storage websites or wish to protect legacy tasks effectively, think about using atmosphere or specs information. These small information document an atmosphere’s contents, permitting you to rebuild it later.

Saving conda environments on this method reduces their dimension on disk from gigabytes to a couple kilobytes. After all, you’ll need to recreate the atmosphere to make use of it. So, you’ll wish to keep away from this method if you happen to steadily revisit tasks that hyperlink to the archived environments.

NOTE: Think about using Mamba, a drop-in substitute for conda, for quicker rebuilds. Because the docs say, “If conda, Mamba!”

Utilizing Surroundings Information: An environmental file is a small file that lists all of the packages and variations put in in an atmosphere, together with these put in utilizing Python’s bundle installer (pip). This helps you each restore an atmosphere and share it with others.

The atmosphere file is written in YAML (.yml), a human-readable data-serialization format for knowledge storage. To generate an atmosphere file, you need to activate after which export the atmosphere. Right here’s the way to make a file for an atmosphere named my_env:

 conda activate my_env
 conda env export > my_env.yml

You may title the file any legitimate filename however watch out as an present file with the identical title will likely be overwritten.

By default, the atmosphere file is written to the consumer listing. Right here’s a truncated instance of the file’s contents:

title: C:Usershannaquick_successfed_hikesfed_env
channels:
  - defaults
  - conda-forge
dependencies:
  - asttokens=2.0.5=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - bottleneck=1.3.4=py310h9128911_0
  - brotli=1.0.9=ha925a31_2
  - bzip2=1.0.8=he774522_0
  - ca-certificates=2022.4.26=haa95532_0
  - certifi=2022.5.18.1=py310haa95532_0
  - colorama=0.4.4=pyhd3eb1b0_0
  - cycler=0.11.0=pyhd3eb1b0_0
  - debugpy=1.5.1=py310hd77b12b_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - entrypoints=0.4=py310haa95532_0

  ------SNIP------

Now you can take away your conda atmosphere and reproduce it once more with this file. To take away an atmosphere, first deactivate it after which run the take away command (the place ENVNAME is the title of your atmosphere):

conda deactivate
conda take away -n ENVNAME --all

If the conda atmosphere exists exterior of Anaconda’s default envs folder, then embody the listing path to the atmosphere, as so:

conda take away -p PATHENVNAME --all

Word that this archiving method will solely work completely if you happen to proceed to make use of the identical working system, reminiscent of Home windows or macOS. It’s because fixing for dependencies can introduce packages that may not be suitable throughout platforms.

To revive a conda atmosphere utilizing a file, run the next, the place my_env represents your conda atmosphere title and atmosphere.yml represents your atmosphere file:

 conda env create -n my_env -f directorypathtoenvironment.yml

It’s also possible to use the atmosphere file to recreate the atmosphere in your D: drive. Simply present the brand new path when utilizing the file. Right here’s an instance:

conda create --prefix D:my_envsmy_new_env --file atmosphere.yml

For extra on atmosphere information, together with the way to manually produce them, go to the docs.

Utilizing Specs Information: If you happen to haven’t put in any packages utilizing pip, you should utilize a specs file to breed a conda atmosphere on the identical working system. To create a specification file, activate an atmosphere, reminiscent of my_env, and enter the next command:

 conda listing --explicit > exp_spec_list.txt

This produces the next output, truncated for brevity:

 # This file could also be used to create an atmosphere utilizing:
 # $ conda create --name  --file 
 # platform: win-64
 @EXPLICIT
 https://conda.anaconda.org/conda-forge/win-64/ca-certificates-202x.xx.x-h5b45459_0.tar.bz2
 https://conda.anaconda.org/conda-forge/noarch/tzdata-202xx-he74cb21_0.tar.bz2

------snip------

Word that the --explicit flag ensures that the focused platform is annotated within the file, on this case, # platform: win-64 within the third line.

Now you can take away the atmosphere as described within the earlier part.

To re-create my_env utilizing this textual content file, run the next with a correct listing path:

conda create -n my_env -f directorypathtoexp_spec_list.txt

4. Archiving Environments with conda-pack

The conda-pack command enables you to archive a conda atmosphere earlier than eradicating it. It packs all the atmosphere right into a compressed archive with the extension: .tar.gz. It’s useful for backing up, sharing, and shifting environments with out the necessity to reinstall packages.

The next command will protect an atmosphere however take away it out of your system (the place my_env represents the title of your atmosphere):

conda set up -c conda-forge conda-pack
conda pack -n my_env -o my_env.tar.gz

To revive the atmosphere later run this command:

mkdir my_env && tar -xzf my_env.tar.gz -C my_env

This system received’t save as a lot reminiscence because the textual content file possibility. Nonetheless, you received’t must re-download packages when restoring an atmosphere, which suggests it may be used with out web entry.

5. Storing Environments on an Exterior Drive

By default, conda shops all environments in a default location. For Home windows, that is underneath the …anaconda3envs folder. You may see these environments by operating the command conda data --envs in a immediate window or terminal. Right here’s the way it appears on my C: drive (it is a truncated view):

Utilizing a Single Environments Folder: In case your system helps an exterior or secondary drive, you’ll be able to configure conda to retailer environments there to release area in your main disk. Right here’s the command; you’ll must substitute your particular path:

conda config --set envs_dirs /path/to/exterior/drive

If you happen to enter a path to your D drive, reminiscent of D:conda_envs, conda will create new environments at this location.

This system works effectively when your exterior drive is a quick SSD and whenever you’re storing packages with massive dependencies, like TensorFlow. The draw back is slower efficiency. In case your OS and notebooks stay on the first drive, you could expertise some learn/write latency when operating Python.

As well as, some OS settings could energy down idle exterior drives, including a delay once they spin again up. Instruments like Jupyter could battle to find conda environments if the drive letter adjustments, so that you’ll wish to use a hard and fast drive letter and make sure that the right kernel paths are set.

Utilizing A number of Surroundings Folders: As a substitute of utilizing a single envs_dirs listing for all environments, you’ll be able to retailer every atmosphere inside its respective venture folder. This allows you to retailer every thing associated to a venture in a single place.

For instance, suppose you will have a venture in your Home windows D: drive in a folder referred to as D:projectsgeospatial. To position the venture’s conda atmosphere on this folder, loaded with ipykernel for JupyterLab, you’d run:

conda create -p D:projectsgeospatialenv ipykernel

After all, you’ll be able to name env one thing extra descriptive, like geospatial_env.

As with the earlier instance, environments saved on a unique disk may cause efficiency points.

Particular Word on JupyterLab: Relying on the way you launch JupyterLab, its default habits could also be to open in your consumer listing (reminiscent of, C:Usersyour_user_name). Since its file browser is restricted to the listing from which it’s launched, you received’t see directories on different drives like D:. There are various methods to deal with this, however one of many easiest is to launch JupyterLab from the D: drive.

For instance, in Anaconda Immediate, kind:

D:

adopted by:

jupyter lab

Now, it is possible for you to to choose from kernels on the D: drive.

For extra choices on altering JupyterLab’s working listing, ask an AI about “the way to change Jupyter’s default working listing” or “the way to create a Symlink to D: in your consumer folder.”

Shifting Current Environments: It is best to by no means manually transfer a conda atmosphere, reminiscent of by reducing and pasting to a brand new location. It’s because conda depends on inside paths and metadata that may grow to be invalid with location adjustments.

As a substitute, it is best to clone present environments to a different drive. This may duplicate the atmosphere, so that you’ll must manually take away it from its authentic location.

Within the following instance, we use the --clone flag to provide a precise copy of a C: drive atmosphere (referred to as my_env) on the D: drive:

conda create -p D:new_envsmy_env --clone C:pathtooldenv

NOTE: Contemplate exporting your atmosphere to a YAML file (as described in Part 3 above) earlier than cloning. This lets you recreate the atmosphere if one thing goes mistaken with the clone process.

Now, whenever you run conda env listing, you’ll see the atmosphere listed in each the C: and D: drives. You may take away the outdated atmosphere by operating the next command within the base atmosphere:

conda take away --name my_env --all -y

Once more, latency points could have an effect on these setups if you happen to’re working throughout two disks.

It’s possible you’ll be questioning, is it higher to maneuver a conda atmosphere utilizing an atmosphere (YAML) file or to make use of--clone? The brief reply is that --clone is the most effective and quickest possibility for shifting an atmosphere to a unique drive on the identical machine. An atmosphere file is greatest for recreating the identical atmosphere on a completely different machine. Whereas the file ensures a constant atmosphere throughout completely different techniques, it may take for much longer to run, particularly with massive environments.

6. Relocating the Bundle Cache

In case your main drive is low on area, you’ll be able to transfer the bundle cache to a bigger exterior or secondary drive utilizing this command:

conda config --set pkgs_dirs D:conda_pkgs

On this instance, packages at the moment are saved on the D drive (D:conda_pkgs) as an alternative of the default location.

If you happen to’re working in your main drive and each drives are SSD, then latency points shouldn’t be important. Nonetheless, if one of many drives is a slower HDD, you’ll be able to expertise slowdowns when creating or updating environments. If D: is an exterior drive linked by USB, you may even see important slowdowns for big environments.

You may mitigate a few of these points by holding the bundle cache (pkgs_dirs) and steadily used environments on the quicker SSD, and different environments on the slower HDD.

One last item to think about is backups. Main drives could have routine backups scheduled however secondary or exterior drives could not. This places you vulnerable to shedding all of your environments.

7. Utilizing Digital Environments

In case your venture doesn’t require conda’s in depth bundle administration system for dealing with heavy dependencies (like TensorFlow or GDAL), you’ll be able to considerably scale back disk utilization with a Python digital atmosphere (venv). This represents a light-weight various to a conda atmosphere.

To create a venv named my_env, run the next command:

One of these atmosphere has a small base set up. A minimal conda atmosphere takes up about 200 MB and contains a number of utilities, reminiscent of conda, pip, setuptools, and so forth. A venv is way lighter, with a minimal set up dimension of solely 5–10 MB.

Conda additionally caches bundle tarballs in pkgs_dirs. These tarballs can develop to a number of GBs over time. As a result of venv installs packages straight into the atmosphere, no additional copies are preserved.

Generally, you’ll wish to take into account venv whenever you solely want primary Python packages like NumPy, pandas, or Scikit-learn. Packages for which conda is strongly really helpful, like Geopandas, ought to nonetheless be positioned in a conda atmosphere. If you happen to use a number of environments, you’ll in all probability wish to follow conda and profit from its bundle linking.

Yow will discover particulars on the way to activate and use Python digital environments within the venv docs.

---

Recap

Excessive impression/low disruption reminiscence administration methods for conda environments embody cleansing the bundle cache and storing little-used environments as YAML or textual content information. These strategies can save many gigabytes of reminiscence whereas retaining Anaconda’s default listing construction.

Different excessive impression strategies embody shifting the bundle cache and/or conda environments to a secondary or exterior drive. This may resolve reminiscence issues however could introduce latency points, particularly if the brand new drive is a sluggish HDD or makes use of a USB connection.

For easy environments, you should utilize a Python digital atmosphere (venv) as a light-weight various to conda.