How to perform CalculiX calculations in parallel

Nowadays most computers have one socket with several cores, allowing for the calculations to be performed in a parallel way. In CalculiX one can

• create the element stiffness matrices in parallel. No special compilation flag is needed. At execution time the environment variable OMP_NUM_THREADS or the environment variable CCX_NPROC_STIFFNESS must be set to the number of cores, default is 1. If both are set, CCX_NPROC_STIFFNESS takes precedence. The maximum number of cores is detected automatically by CalculiX by using the sysconf(_SC_NPROCESSORS_CONF) function. It can be overriden by the user by means of environment variable NUMBER_OF_CPUS.

  Notice that older GNU-compiler versions (e.g. gcc 4.2.1) may have problems with this parallellization due to the size of the fields to be allocated within each thread (e.g. s(100,100) in routine e_3d.f). This should not be a problem with the actual compiler version.

• solve the system of equations with the multithreaded version of SPOOLES. To this end
  • the MT-version of SPOOLES must have been compiled. For further information on this topic please consult the SPOOLES documentation
  • CalculiX CrunchiX must have been compiled with the USE_MT flag activated in the Makefile, please consult the README.INSTALL file.
  • at execution time the environment variable OMP_NUM_THREADS must have been set to the number of cores you want to use. In Linux this can be done by “export OMP_NUM_THREADS=n” on the command line, where n is the number of cores. Default is 1. Alternatively, you can set the number of cores using the environment variable CCX_NPROC_EQUATION_SOLVER. If both are set, the latter takes precedence.

• solve the system of equations with the multithreaded version of PARDISO. PARDISO is proprietary. Look at the PARDISO documentation how to link the multithreaded version. At execution time the environment variable OMP_NUM_THREADS must be set to the number of cores, default is 1.

• create material tangent matrices and calculate the stresses at the integration points in parallel. No special compilation flag is needed. At execution time the environment variable OMP_NUM_THREADS or the environment variable CCX_NPROC_RESULTS must be set to the number of cores, default is 1. If both are set, CCX_NPROC_RESULTS takes precedence. The maximum number of cores is detected automatically by CalculiX by using the sysconf(_SC_NPROCESSORS_CONF) function. It can be overriden by the user by means of environment variable NUMBER_OF_CPUS. Notice that if a material user subroutine (Sections 8.5 and 8.6) is used, certain rules have to be complied with in order to allow parallelization. These include (this list is possibly not exhaustive):
  • no save statements
  • no data statements
  • avoid logical variables
  • no write statements

• calculate the viewfactors for thermal radiation computations in parallel. No special compilation flag is needed. At execution time the environment variable OMP_NUM_THREADS or the environment variable CCX_NPROC_VIEWFACTOR must be set to the number of cores, default is 1. If both are set, CCX_NPROC_VIEWFACTOR takes precedence. The maximum number of cores is detected automatically by CalculiX by using the sysconf(_SC_NPROCESSORS_CONF) function. It can be overriden by the user by means of environment variable NUMBER_OF_CPUS.

• perform several operations in CFD calculations (computational fluid dynamics) in parallel. No special compilation flag is needed. At execution time the environment variable OMP_NUM_THREADS or the environment variable CCX_NPROC_CFD must be set to the number of cores, default is 1. If both are set, CCX_NPROC_CFD takes precedence. The maximum number of cores is detected automatically by CalculiX by using the sysconf(_SC_NPROCESSORS_CONF) function. It can be overriden by the user by means of environment variable NUMBER_OF_CPUS.

• Calculate the magnetic intensity by use of the Biot-Savart law in parallel. No special compilation flag is needed. At execution time the environment variable OMP_NUM_THREADS or the environment variable CCX_NPROC_BIOTSAVART must be set to the number of cores, default is 1. If both are set, CCX_NPROC_BIOTSAVART takes precedence. The maximum number of cores is detected automatically by CalculiX by using the sysconf(_SC_NPROCESSORS_CONF) function. It can be overriden by the user by means of environment variable NUMBER_OF_CPUS.

• Perform several vector and matrix operations needed by the SLATEC iterative solvers or by ARPACK in parallel. To this end the user must have defined the environment variable OMP_NUM_THREADS, and used the openmp FORTRAN flag in the Makefile. The parallellization is done in FORTRAN routines using openmp. The corresponding lines start with “c$omp”. If the openmp flag is not used, these lines are interpreted by the compiler as comment lines and no parallellization takes place. Notice that this parallellization only pays off for rather big systems, let's say 300,000 degrees of freedom for CFD-calculations or 1,000,000 degrees of freedom for mechanical frequency calculations.

Examples:

• For some reason the function sysconf does not work on your computer system and leads to a segmentation fault. You can prevent using the function by defining the maximum number of cores explicitly using the NUMBER_OF_CPUS environment variable
• You want to perform a thermomechanical calculation, but you are using a user defined material subroutine (Sections 8.5 and 8.6) which is not suitable for parallelization. You can make maximum use of parallelization (e.g. for the calculation of viewfactors) by setting the variable OMP_NUM_THREADS to the maximum number of cores on your system, and prevent parallelization of the material tangent and stress calculation step by setting CCX_NPROC_RESULTS to 1.