How to compile and install RELION-3.1 on CentOS 8.1 • Guillaume (2024)

2022-05-10 update: this post is now mostly outdated. The general strategy ofinstalling in /opt and using module files is still relevant, but the specificdetails and versions mentioned below are outdated. In addition, CentOStransitioned to anupdate schedule that is out of sync with the Nvidia driver. I now use RockyLinux for cryoEM computing, and I also no longerrecommend installing the Nvidia driver with the “runfile” installer: it is mucheasier to use the RPM package provided by Nvidia (the CUDA toolkit, however, isbest installed in /opt using the “runfile”). You can find up-to-dateinstallation instructions for RELION here.

The lab I work at recently acquired a GPU workstation on which I had to installRELION, a program for processing cryoEM data. Sincethis is not a straightforward procedure, I took some notes in case I need to dothis again in the future. I decided to also post these notes here, in case theycan help anyone else.

These notes apply to RELION-3.1 and CentOS 8.1. Usual disclaimers apply: backupyour data before modifying your system, don’t run commands you don’t fullyunderstand (especially so if they require to be run with sudo), follow thesedirections at your own risk, and I am in no way responsible if you mess up yoursystem in the course of following these directions. Also, I make no commitmentto keep these notes up-to-date with future versions of RELION or CentOS, and Idon’t have time to offer individual help. So please don’t email me questions,email the CCPEM list instead for questions specific to RELION, orseek help from your distribution’s specific channels for general Linux questions.

Conventions

All commands listed below should be run as a normal user, not as root.Commands that require administrator permissions (like installing packages withthe system package manager) are prepended with sudo, which assumes the useraccount you run these commands from is in the administrator group (wheel).

Programs not installed with the system package manager will be installed intheir own directory /opt/<program>-X.Y.Z, in which <program> is theprogram’s name in lowercase and X.Y.Z is its version number. This will onlywork if the user performing the installation has write permission to /opt,which is safe to do because /opt is a location reserved for programs that arenot part of the system distribution (there is nothing to break there,because the system doesn’t put any of its files there). This has severaladvantages over making a package for the distribution or installing in/usr/local:

  • the biggest advantage is that we can have several versions of the same programinstalled at the same time: we can only have one relion binary under/usr/local, but we can have several /opt/relion-X.Y.Z with differentversion numbers happily living next to each other (sometimes, we need torevisit old results obtained with an earlier version of the program notnecessarily compatible with the current version, so this is a true practicaladvantage);
  • installing a program is easy: in most cases, ./configure --prefix=/opt/<program>-X.Y.Z ; make ; make install will work just fine (asindicated above, without any risk of messing up the base system if run as aregular user), which is easier than making an RPM or DEB or what-have-youpackage compliant with all of your distribution’s packaging rules;
  • it is easy to check what takes up storage space, with du -sh /opt/*;
  • uninstalling a program is as easy as rm -r /opt/<program>-X.Y.Z, since allof a program’s files are under a single directory, instead of scattered acrosssubdirectories under /usr/local.

Now, /usr/local/bin is in users’ PATH by default, while this is not the casefor arbitrary directories under /opt. How do we make our custom-built programsaccessible from the shell with minimal configuration for our users? Obviously,having every user edit their ~/.bashrc file is not a viable option: they wouldhave to do that every time they want to change which version of a program theyuse, and this is error prone. The solution is to use the EnvironmentModules system, installed by default on CentOS 8.1. This allows us towrite modulefiles that will correctly set up environment variables for eachspecific program. We will store these modulefiles under /opt/modulefiles, andappend this path to $MODULESHOME/init/.modulespath so the module commandscan use our custom modulefiles. The file $MODULESHOME/init/.modulespathinitially looks something like this:

# This file defines the initial setup for the modulefiles search path# Each line containing one or multiple paths delimited by ':' will be# added to the MODULEPATH environment variable./usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles

Edit it with sudo vi $MODULESHOME/init/.modulespath to add /opt/modulefilesto the $MODULEPATH variable. The file should now look like this:

# This file defines the initial setup for the modulefiles search path# Each line containing one or multiple paths delimited by ':' will be# added to the MODULEPATH environment variable./usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/modulefiles

Our users can now list all available modules on the system (module avail)and easily set their shell environment to use what they need(module load <program/X.Y.Z>).

Requirements

To compile RELION, we need to install development tools:

sudo dnf group install "Development Tools"sudo dnf install cmake

If you wonder what comes with the “Development Tools” group, you can find outwith the following command:

dnf group info "Development Tools"

And we need the following libraries:

sudo dnf install fftw-devel fltk-devel libX11-devel libtiff-devel libpng-devel freetype-devel

The following may not be necessary for RELION, but some libraries are in thePowerTools repository, so we might need to activate it:

sudo dnf config-manager --set-enabled PowerTools

And other required packages are in EPEL:

sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

Choose a version of CUDA

Compiling and running RELION requires the CUDA Toolkit and libraries. Theeasiest way to compile RELION-3.1 on CentOS 8.1 is to use CUDA 10.2, whichsupports the version of GCC (8.3.1) that comes with CentOS 8.1. However, RELIONcan use external programs for motion correction and CTF estimation, which arenot open source and depend on different versions of CUDA. The trade-offs, at thetime I am writing this, go as follows:

  1. Compile RELION with CUDA 10.2. This is the easiest way to go and has theleast number of pre-requisites. For motion correction, you will be able touse either MotionCor2 (not open source, but version 1.3.1 ships abinary compiled with CUDA 10.2) or RELION’s own motion correction program(not GPU-accelerated). For CTF estimation, you will be limited toCTFFIND4 (not GPU-accelerated), since Gctf version 1.18does not ship a binary compiled with CUDA 10.2 (and likely won’t because itseems not maintained anymore; it is also not open source, preventing one fromcompiling it with one’s preferred version of CUDA).
  2. Compile RELION with CUDA 9.2. This is a bit more difficult because CUDA9.2 is not compatible with GCC 8.3.1, which is the default compiler on CentOS8.1: for CUDA 9.2 to work, you will therefore have to install a version ofGCC earlier than version 7. The advantage of using CUDA 9.2 is that you canthen have all programs working in the same environment: MotionCor2 (version1.3.1 ships a binary built with CUDA 9.2), Gctf (version 1.18 also ships abinary built with CUDA 9.2), RELION’s own motion correction program(independent of CUDA because not GPU-accelerated) and CTFFIND4 (alsoindependent of CUDA because not GPU-accelerated). Even though this is a bitmore work for the system administrator, this is an easier setup for the userssince they will be able to use any combination of these programs in a singleshell.
  3. Compile two copies of RELION, one with CUDA 9.2 and one with CUDA 10.2.This is as easy (or as difficult…) as both options above, but possible whenusing the environment modules system. This way, one can use the RELIONcompiled with CUDA 10.2 for everything, and only use the one compiled withCUDA 9.2 to run Gctf. This requires changing the environment to choose whichversion of RELION and CUDA should be used at run time, but this is easywith the environment modules system.

I wanted to go with option 1, but the CTFFIND4 binaries downloaded from itswebsite don’t run on CentOS 8.1 (they crash with a segmentation fault, andinspecting them with file ctffind reports they were compiled for a Linux 2.6kernel, which is several versions older than the kernel version in CentOS 8.1).Since this program is open source, I tried to compile it. It compiled fine andruns long enough to interactively pass all parameter prompts, but then crasheswith a segmentation fault. This may be due to the fact that I compiled it withGCC, as a somewhat old email on the CCPEM list suggests that it shouldbe compiled with ICC. So I chose option 3.

Install CUDA 10.2

The following commands will download the installer fromNvidia’s website and install only the CUDA Toolkit (not the driver):

cd ~/Downloadswget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.runchmod +x cuda_10.2.89_440.33.01_linux.run./cuda_10.2.89_440.33.01_linux.run --silent --toolkit --toolkitpath=/opt/cuda-10.2

Then place this module file in /opt/modulefiles/cuda/10.2:

#%Module1.0 proc ModulesHelp { } {global dotversionputs stderr "\tCUDA Libraries and Toolkit, version 10.2"}module-whatis "CUDA Libraries and Toolkit. Documentation: https://docs.nvidia.com/cuda/archive/10.2/"conflict cudaset program cudaset version 10.2set prefix /opt/$program-$versionprepend-path PATH $prefix/binprepend-path LD_LIBRARY_PATH $prefix/lib64prepend-path CPATH $prefix/includeprepend-path C_INCLUDE_PATH $prefix/includeprepend-path CPLUS_INCLUDE_PATH $prefix/includeprepend-path INCLUDE $prefix/includesetenv CUDA_HOME $prefix

Install CUDA 9.2

For this, we first need to install a version of GCC earlier than version 7. Ichose to go for the latest version in the 6.x series, which is 6.5.0. CompilingGCC requires the following system packages:

sudo dnf install gmp-devel mpfr-devel libmpc-devel

And here is how to download GCC 6.5.0’s source, compile and install it (you canof course choose a different mirror closer to you):

cd ~/Downloadswget ftp://ftp.uvsq.fr/pub/gcc/releases/gcc-6.5.0/gcc-6.5.0.tar.gztar -xf gcc-6.5.0.tar.gzcd gcc-6.5.0mkdir buildcd build../configure --prefix=/opt/gcc-6.5.0 --disable-multilibmakemake install-strip

Finally, save this module file as /opt/modulefiles/gcc/6.5.0:

#%Module1.0 proc ModulesHelp { } {global dotversionputs stderr "\tGNU Compiler Collection, version 6.5.0"}module-whatis "GNU Compiler Collection. Documentation: https://www.gnu.org/software/gcc/"set program gccset version 6.5.0set prefix /opt/$program-$versionprepend-path PATH $prefix/binprepend-path LD_LIBRARY_PATH $prefix/libprepend-path LD_LIBRARY_PATH $prefix/lib64prepend-path CPATH $prefix/includeprepend-path C_INCLUDE_PATH $prefix/includeprepend-path CPLUS_INCLUDE_PATH $prefix/include# Take higher priority than system CCsetenv CC $prefix/bin/gccsetenv CXX $prefix/bin/g++

We can then install CUDA 9.2:

cd ~/Downloadswget https://developer.nvidia.com/compute/cuda/9.2/Prod2/local_installers/cuda_9.2.148_396.37_linuxchmod +x cuda_9.2.148_396.37_linux./cuda_9.2.148_396.37_linux --silent --toolkit --toolkitpath=/opt/cuda-9.2

Let’s also install the patch:

module purgemodule load gcc/6.5.0cd ~/Downloadswget https://developer.nvidia.com/compute/cuda/9.2/Prod2/patches/1/cuda_9.2.148.1_linuxchmod +x cuda_9.2.148.1_linux./cuda_9.2.148.1_linux --silent --accept-eula --installdir=/opt/cuda-9.2

And finally, save this module file as /opt/modulefiles/cuda/9.2:

#%Module1.0 proc ModulesHelp { } {global dotversionputs stderr "\tCUDA Libraries and Toolkit, version 9.2"}module-whatis "CUDA Libraries and Toolkit. Documentation: https://docs.nvidia.com/cuda/archive/9.2/"conflict cudaset program cudaset version 9.2set prefix /opt/$program-$versionprepend-path PATH $prefix/binprepend-path LD_LIBRARY_PATH $prefix/lib64prepend-path CPATH $prefix/includeprepend-path C_INCLUDE_PATH $prefix/includeprepend-path CPLUS_INCLUDE_PATH $prefix/includeprepend-path INCLUDE $prefix/includesetenv CUDA_HOME $prefix

Install OpenMPI 3.1.6

RELION also requires OpenMPI. I first tried to use the RPM package:

sudo dnf install openmpi-devel

I managed to compile RELION with this OpenMPI, which happens to be version4.0.1, but then I got segmentation faults at run time, making this build ofRELION essentially useless (parallelization with OpenMPI is used for prettymuch everything in RELION). I investigated this error and found that cmake hadpicked up an OpenMPI version 3.1:

-- Found MPI_C: /usr/lib64/openmpi/lib/libmpi.so (found version "3.1")

I still don’t understand how this is possible, since the RPM package did notinstall anything from version 3.1:

$ ls -l /usr/lib64/openmpi/lib/ | grep libmpi.solrwxrwxrwx. 1 root root 17 Nov 21 2019 libmpi.so -> libmpi.so.40.20.1lrwxrwxrwx. 1 root root 17 Nov 21 2019 libmpi.so.40 -> libmpi.so.40.20.1-rwxr-xr-x. 1 root root 2422280 Nov 21 2019 libmpi.so.40.20.1

But then the mpirun used at run time was definitely 4.0.1, and that causedproblems:

$ mpirun --versionmpirun (Open MPI) 4.0.1

I asked about this problem on the CCPEM list, and from the answerI got I understand that using different versions of OpenMPI duringcompilation and at run time definitely cause this kind of problem. But what isstill not clear to me is whether RELION would have worked fine with OpenMPI4.0.1, had it been correctly compiled and run with this version, or whether itis only compatible with OpenMPI 3.x versions. So, I installed OpenMPI 3.1.6(newest version in the 3.x series) from source:

cd ~/Downloadswget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.6.tar.bz2tar -xf openmpi-3.1.6.tar.bz2cd openmpi-3.1.6mkdir buildcd build../configure --prefix=/opt/openmpi-3.1.6makemake install-strip

I then adapted the module file provided by the RPM package to make one for thisversion of OpenMPI. I stored this module file in/opt/modulefiles/openmpi/3.1.6:

#%Module 1.0## OpenMPI module for use with 'environment-modules' package:#conflict mpiprepend-path PATH /opt/openmpi-3.1.6/binprepend-path LD_LIBRARY_PATH /opt/openmpi-3.1.6/libprepend-path PKG_CONFIG_PATH /opt/openmpi-3.1.6/lib/pkgconfigprepend-path MANPATH /opt/openmpi-3.1.6/share/mansetenv MPI_BIN /opt/openmpi-1.3.6/binsetenv MPI_SYSCONFIG /opt/openmpi-3.1.6/etcsetenv MPI_FORTRAN_MOD_DIR /usr/lib64/gfortran/modules/openmpisetenv MPI_INCLUDE /opt/openmpi/3.1.6/includesetenv MPI_LIB /opt/openmpi-3.1.6/libsetenv MPI_MAN /opt/openmpi-3.1.6/share/mansetenv MPI_PYTHON_SITEARCH /usr/lib64/python3.6/site-packages/openmpisetenv MPI_PYTHON2_SITEARCH /usr/lib64/python3.6/site-packages/openmpisetenv MPI_PYTHON3_SITEARCH /usr/lib64/python3.6/site-packages/openmpisetenv MPI_COMPILER openmpi-x86_64setenv MPI_SUFFIX _openmpisetenv MPI_HOME /opt/openmpi-3.1.6

Install RELION

After many detours, we finally have all we need to compile and install RELION.We get the source code by cloning the git repository the firsttime:

mkdir ~/softwarecd ~/softwaregit clone https://github.com/3dem/relion

Next time you need to update it, pull the new changes:

cd ~/software/reliongit pull

Compile RELION with CUDA 10.2

The following commands will configure and compile RELION-3.1 (at the time Iwrote these notes, it was commit 5997001f75) with CUDA 10.2.Change the value of -DCUDA_ARCH= to adapt to your GPU: this is the “computecapability” listed here, without the dot (choosethe highest one supported by both your GPU and the version of CUDA you’re using;if you don’t specify this option, cmake seems to default to a very low computecapability, which is compatible with more different combinations of GPU andCUDA version but means you won’t take full advantage of your specific GPU):

cd ~/software/reliongit checkout ver3.1mkdir build_cuda-10.2cd build_cuda-10.2module purgemodule load openmpi/3.1.6 cuda/10.2cmake -DCMAKE_INSTALL_PREFIX=/opt/relion-3.1_cuda-10.2 -DCUDA_ARCH=75 ..make

To install it, make sure the destination directory exists:

mkdir /opt/relion-3.1_cuda-10.2

And then run:

make install

Finally, save this module file as /opt/modulefiles/relion/3.1_cuda-10.2. Thereare more environment variables you can set based on your specific system, youcan read about it in RELION’s documentation.

#%Module1.0proc ModulesHelp { } {global dotversionputs stderr "\tRELION, version 3.1 (CUDA 10.2)"}module-whatis "2D classification, 3D classification and 3D refinement. Documentation: https://www3.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page"module load openmpi/3.1.6 cuda/10.2 motioncor2/1.3.1 ctffind/4.1.14conflict relionprereq openmpi/3.1.6prereq cuda/10.2prereq motioncor2prereq ctffindset program relionset version 3.1_cuda-10.2set prefix /opt/$program-$version# Where to find other programssetenv RELION_MOTIONCOR2_EXECUTABLE MotionCor2_v1.3.1-Cuda102setenv RELION_CTFFIND_EXECUTABLE ctffindsetenv RELION_RESMAP_EXECUTABLE ResMapsetenv RELION_PDFVIEWER_EXECUTABLE evincesetenv RELION_QSUB_TEMPLATE $prefix/bin/qsub.csh# MPI and threads settings# Ask for confirmation if users try to submit local jobs with more than 9 MPI processes. Rationale: 9 MPIs means 1 coordinator + 4GPUs x 2 workers.setenv RELION_WARNING_LOCAL_MPI 9# It doesn't help to overbook the GPUs too much. 13 MPIs means 1 coordinator + 4GPUs x 3 workers.# But some programs like CTFFIND and RELION's MotionCor run on CPUs, so the hard limit on MPI processes should be half the CPU cores.setenv RELION_MPI_MAX 40setenv RELION_ERROR_LOCAL_MPI 41# Shell to launch other programs fromsetenv RELION_SHELL bash# Scratch locationsetenv RELION_SCRATCH_DIR /scratch/prepend-path PATH $prefix/binprepend-path LD_LIBRARY_PATH $prefix/lib

Compile RELION with CUDA 9.2

The following commands will configure and compile RELION-3.1 (at the time Iwrote these notes, it was commit 5997001f75) with CUDA 9.2.Change the value of -DCUDA_ARCH= to adapt to your GPU: this is the “computecapability” listed here, without the dot (choosethe highest one supported by both your GPU and the version of CUDA you’re using;if you don’t specify this option, cmake seems to default to a very low computecapability, which is compatible with more different combinations of GPU andCUDA version but means you won’t take full advantage of your specific GPU):

cd ~/software/reliongit checkout ver3.1mkdir build_cuda-9.2cd build_cuda-9.2module purgemodule load openmpi/3.1.6 cuda/9.2 gcc/6.5.0cmake -DCMAKE_INSTALL_PREFIX=/opt/relion-3.1_cuda-9.2 -DCUDA_ARCH=72 ..make

To install it, make sure the destination directory exists:

mkdir /opt/relion-3.1_cuda-9.2

And then run:

make install

Finally, save this module file as /opt/modulefiles/relion/3.1_cuda-9.2. Thereare more environment variables you can set based on your specific system, youcan read about it in RELION’s documentation.

#%Module1.0 proc ModulesHelp { } {global dotversionputs stderr "\tRELION, version 3.1 (CUDA 9.2)"}module-whatis "2D classification, 3D classification and 3D refinement. Documentation: https://www3.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page"module load openmpi/3.1.6 cuda/9.2 motioncor2/1.3.1 gctf/1.18b2conflict relionprereq openmpi/3.1.6prereq cuda/9.2prereq motioncor2prereq gctf#prereq ctffindset program relionset version 3.1_cuda-9.2set prefix /opt/$program-$version# Where to find other programssetenv RELION_MOTIONCOR2_EXECUTABLE MotionCor2_v1.3.1-Cuda92setenv RELION_GCTF_EXECUTABLE Gctf_v1.18_b2_sm70_cu9.2#setenv RELION_CTFFIND_EXECUTABLE ctffindsetenv RELION_RESMAP_EXECUTABLE ResMapsetenv RELION_PDFVIEWER_EXECUTABLE evincesetenv RELION_QSUB_TEMPLATE $prefix/bin/qsub.csh# MPI and threads settings# Ask for confirmation if users try to submit local jobs with more than 9 MPI processes. Rationale: 9 MPIs means 1 coordinator + 4GPUs x 2 workers.setenv RELION_WARNING_LOCAL_MPI 9# It doesn't help to overbook the GPUs too much. 13 MPIs means 1 coordinator + 4GPUs x 3 workers.# But some programs like CTFFIND, Gctf, MotionCor2 and RELION's MotionCor run on CPUs, so the hard limit on MPI processes should be half the CPU cores.setenv RELION_MPI_MAX 40setenv RELION_ERROR_LOCAL_MPI 41# Shell to launch other programs fromsetenv RELION_SHELL bash# Scratch locationsetenv RELION_SCRATCH_DIR /scratch/prepend-path PATH $prefix/binprepend-path LD_LIBRARY_PATH $prefix/lib

Running RELION

Now, the command module avail should list all these new module files (I didnot show the module files for MotionCor2, Gctf and CTFFIND; those simply containthe same header up to module-whatis, followed by a single prepend-pathdirective to indicate where to find the binary):

$ module avail---/opt/modulefiles ---cuda/9.2cuda/10.2ctffind/4.1.14gcc/6.5.0gctf/1.18b2motioncor2/1.3.1openmpi/3.1.6relion/3.1_cuda-9.2relion/3.1_cuda-10.2 

You can now get one version of RELION or the other on your path like so:

$ module listNo Modulefiles Currently Loaded.$ which relion/usr/bin/which: no relion in (/home/guillaume/.local/bin:/home/guillaume/bin:/opt/miniconda3/condabin:/home/guillaume/.local/bin:/home/guillaume/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)$ module load relion/3.1_cuda-9.2$ module listCurrently Loaded Modulefiles: 1) openmpi/3.1.6 2) cuda/9.2 3) motioncor2/1.3.1 4) gctf/1.18b2 5) relion/3.1_cuda-9.2$ which relion/opt/relion-3.1_cuda-9.2/bin/relion$ module purge$ module listNo Modulefiles Currently Loaded.$ module load relion/3.1_cuda-10.2$ module listCurrently Loaded Modulefiles: 1) openmpi/3.1.6 2) cuda/10.2 3) motioncor2/1.3.1 4) ctffind/4.1.14 5) relion/3.1_cuda-10.2$ which relion/opt/relion-3.1_cuda-10.2/bin/relion
How to compile and install RELION-3.1 on CentOS 8.1 • Guillaume (2024)

References

Top Articles
Latest Posts
Article information

Author: Jonah Leffler

Last Updated:

Views: 6177

Rating: 4.4 / 5 (65 voted)

Reviews: 88% of readers found this page helpful

Author information

Name: Jonah Leffler

Birthday: 1997-10-27

Address: 8987 Kieth Ports, Luettgenland, CT 54657-9808

Phone: +2611128251586

Job: Mining Supervisor

Hobby: Worldbuilding, Electronics, Amateur radio, Skiing, Cycling, Jogging, Taxidermy

Introduction: My name is Jonah Leffler, I am a determined, faithful, outstanding, inexpensive, cheerful, determined, smiling person who loves writing and wants to share my knowledge and understanding with you.