.. _building-dane: Dane (LLNL) ============= The `Dane Intel CPU cluster `__ is located at LLNL. Introduction ------------ If you are new to this system, **please see the following resources**: * `LLNL user account `__ (login required) * `Jupyter service `__ (`documentation `__, login required) * `Production directories `__: * ``/p/lustre1/$(whoami)`` and ``/p/lustre2/$(whoami)``: personal directory on the parallel filesystem * Note that the ``$HOME`` directory and the ``/usr/workspace/$(whoami)`` space are NFS mounted and *not* suitable for production quality data generation. .. _building-dane-preparation: Preparation ----------- Use the following commands to download the WarpX source code. Note that these commands and the shell scripts all assume the bash shell. This downloads WarpX into the workspace directory, which is recommended. WarpX can be downloaded elsewhere if that doesn't work with your directory structure, but note that the commands shown below refer to WarpX in the workspace directory. .. code-block:: bash git clone https://github.com/BLAST-WarpX/warpx.git /usr/workspace/${USER}/dane/src/warpx The system software modules, environment hints, and further dependencies are setup via the file ``$HOME/dane_warpx.profile`` which is copied from the WarpX source. Set it up now: .. code-block:: bash cp /usr/workspace/${USER}/dane/src/warpx/Tools/machines/dane-llnl/dane_warpx.profile.example $HOME/dane_warpx.profile .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: ../../../../Tools/machines/dane-llnl/dane_warpx.profile.example :language: bash Edit the 2nd line of this script, which sets the ``export proj=""`` variable. For example, if you are member of the project ``tps``, then run ``vi $HOME/dane_warpx.profile``. Enter the edit mode by typing ``i`` and edit line 2 to read: .. code-block:: bash export proj="tps" Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). .. important:: Now, and as the first step on future logins to Dane, activate these environment settings by executing the file: .. code-block:: bash source $HOME/dane_warpx.profile Finally, since Dane does not yet provide software modules for some of our dependencies, WarpX provides a script to install them. This is done executed now. They are by default installed in the workspace directory (which is recommended), but can be installed elsewhere by setting the environment variable ``WARPX_SW_DIR``. The second command activates the Python virtual environment. This would normally be done by the ``dane_warpx.profile`` script, but the environment is created by the install script and so wasn't created yet when the profile was run above. So the activation needs to be done this way only this one time. .. code-block:: bash bash /usr/workspace/${USER}/dane/src/warpx/Tools/machines/dane-llnl/install_dependencies.sh source /usr/workspace/${USER}/dane/venvs/warpx-dane/bin/activate .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down .. literalinclude:: ../../../../Tools/machines/dane-llnl/install_dependencies.sh :language: bash .. _building-dane-compilation: Compilation ----------- Use the following :ref:`cmake commands ` to compile the application executable. The options should be modified to suit your needs, for example only building for the dimensions needed. .. code-block:: bash cd /usr/workspace/${USER}/dane/src/warpx rm -rf build_dane cmake -S . -B build_dane -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" cmake --build build_dane -j 6 The WarpX application executables are now in ``/usr/workspace/${USER}/dane/src/warpx/build_dane/bin/``. Additionally, the following commands will install WarpX as a Python module: .. code-block:: bash rm -rf build_dane_py cmake -S . -B build_dane_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3" cmake --build build_dane_py -j 6 --target pip_install Now, you can :ref:`submit Dane compute jobs ` for WarpX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). Or, you can use the WarpX executables to submit Dane jobs (:ref:`example inputs `). For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$PROJWORK/$proj/``. .. _building-dane-update: Update WarpX & Dependencies --------------------------- If you already installed WarpX in the past and want to update it, start by getting the latest source code: .. code-block:: bash cd /usr/workspace/${USER}/dane/src/warpx # read the output of this command - does it look ok? git status # get the latest WarpX source code git pull # read the output of these commands - do they look ok? git status git log # press q to exit And, if needed, - :ref:`update the dane_warpx.profile file `, - log out and into the system, activate the now updated environment profile as usual, - :ref:`execute the dependency install scripts `. As a last step, clean the build directory ``rm -rf /usr/workspace/${USER}/dane/src/warpx/build_dane`` and rebuild WarpX. .. _running-cpp-dane: Running ------- .. _running-cpp-dane-CPUs: Intel Sapphire Rapids CPUs ^^^^^^^^^^^^^^^^^^^^^^^^^^ The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Dane at LLNL. Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``plasma_mirror_inputs``. .. literalinclude:: ../../../../Tools/machines/dane-llnl/dane.sbatch :language: bash :caption: You can copy this file from ``Tools/machines/dane-llnl/dane.sbatch``. To run a simulation, copy the lines above to a file ``dane.sbatch`` and run .. code-block:: bash sbatch dane.sbatch to submit the job.