This is the benchmark package for the NCEP ufs-weather-model GFSV17:P8. This benchmark consists of the source and dependencies for ufs-weather-model and gsi. THere are testcases for c384atm, c1152atm, c768coupled, and DA (gsi) These are in separate tarballs but use the executables from this package. c384atm c1152adm and c768coupled untar with this package. This allowed the jobs to use relative path names. DA is a separate tarball. All use the relative path name ../BENCH.SRC.EXEC for executables in their jobs. The build scripts build atm-only and coupled ufs-weather-model and the gsi.x executable dependences for ufs-weather-model are included. Benchmarkers and testers should not need admin or third party built modules for any dependency except compilers ,python (3.6 minimum), cmake(3.19 minimum) and MPI. Dependencies should be built before the forecast model, then the forecast models are built, and then they can be run. Currently dependencies are supplied by the simple-ufs libraries included in this distribution. In the latter case no modules are used, instead environment variables are just set in the model build shell Steps 0. Load your intel compiler, python 3 and MPI environment. These are not provided. Cmake must be at level 3.19 or higher 0a. Set environment FC, CC and CXX to appropriate MPI fortran C and C++ compilers 0c. (only needed with multiple build attempts.. the initial untarred distribution won't have these) run ./clean to remove all libraries, objects, and executables and source trees At this point the script MAKEEVERYTHING.sh will build all dependencies, the ufs-weather-model and the GSI executable. Below is a deccription of what it does or the steps if you want to separate the build into steps. 1. cd to ./DEPENDENCIES tar -xvf SIMPLE-LIBS.TAR cd ./simple-ufs sh -x ../BUILD.SH intel -ufslibs #*********************************************************************************** At the end of this all third party dependencies for ufs-weather-model should be available. The script ./simple-ufs/envset sets the environment variables needed for the model builds. If this script is not present the build of the dependencies has failed. 2. Build the executables cd .. (the root of this distribution) tar -xvf UFS.WEATHER.MODEL.TAR cd to ./ufs-weather-model source ../DEPENDENCIES/simple-ufs/envset rm -rf build mkdir build cd build # load or get a cmake that is at least at 3.19.x and a python that is at least 3.6 # build COUPLED model cmake .. -DAPP=S2SWA -DCCPP_SUITES=FV3_GFS_v17_coupled_p8,FV3_GFS_cpld_rasmgshocnsstnoahmp_ugwp \ -D32BIT=ON \ -DINLINE_POST=ON \ -DPARALLEL_NETCDF=ON \ -DCMAKE_INSTALL_PREFIX=install make -j 8 cp ufs_model ../exec/ufs.coupled.x # build ATM ONLY model cd .. rm -rf build mkdir build cd build cmake .. -DAPP=S2SWA -DCCPP_SUITES=FV3_GFS_v17_p8 \ -D32BIT=ON \ -DINLINE_POST=ON \ -DPARALLEL_NETCDF=ON \ -DCMAKE_INSTALL_PREFIX=install make -j 8 cp ufs_model ../exec/ufs.atm.x 3. Run the model Run directories for the cases are c384atm and c1152atm and c768coupled Job job.wcoss2 in these directories runs the model on wcoss2. You may need to change the batch scheduler directives, and environment variables on other systems. All paths in these directories are relative e.g. the executables and prebuilt libraries, so there should be no dependencies on the WCOSS2 local environment. All case directories also have a jbase file which is used to create a new job if resources are changed Changing MPI ranks is described next. 4 change resources (draft) The number of mpi ranks needed is 6 * xlayout * ylayout + IO group size * IO group number an example relevant line is in ./c384atm/input.nml "layout = 4,4" so 96 integration tasks are needed. There is one write group specified in these lines of model_configure write_groups: 1 write_tasks_per_group: 32 1*32 is 32 . Add 96 to get 128 total MPI tasks. MPI tasks can be increased by increasing x or y or both layouts in ./input.nml. 8 and 12 are known to work for c384. 12, 16, 24, 32 , 36, 48 and 64 are known to work for c1152 Others depend on an integral factor of "something" rule and have not been evaluated at this writing. A script ./decompose xd yd procs.per.group ngroups nocean nice 0 text.label is provided to change the decomposition in each runcase. In ATM only cases nocean and nice are zero. This application scales well with threading . OMP_NUM_THREADS can be increased to 4 to 8, possibly larger. No change is needed in model configuration files if threading is changed. The configuration provided for c384atm specifies minimal resources. This is intended to be run in ensemble mode with large numbers of copies and minimal scalability needed. ./C1152tm requires much larger resources and is approximately 6-8x more expensive to run. scripts below may prove handy in lieu of chopping up the MAKEEVERYTHING script script makeall.sh builds the dependency libraries and then builds the ufs executable. script ufs.make.sh builds the ufs-weather-model only and assumes dependencies are already built 5 verification A python script to read three different forecast files for comparison is provided in ./verify/cmp.py. This script requires cftime >= 1.3.0 Cython >= 0.29.21 numpy >= 1.19.4 pandas >= 1.1.5 python-dateutil >= 2.8.1 h5py >= 3.1.0 netcdf4 >= 1.5.5.1 The following installs it ## setup python, usually module load python.something cd verify cat <ufswm_requirements.txt cftime >= 1.3.0 Cython >= 0.29.21 numpy >= 1.19.4 pandas >= 1.1.5 python-dateutil >= 2.8.1 h5py >= 3.1.0 netcdf4 >= 1.5.5.1 EOF python -m venv ufswm source ufswm/bin/activate pip install -r ufswm_requirements.txt cd - The following runs it cd [c384atm c768coupled c1152atm ] # one of these three sh -x ../verify.sh This will produce a sfcf024_butterfly_test.log file in your cwd Inspect this for results. It compares the ratio of a tiny perturbation difference after 24 hours to the ratio of your forecast differences after 24 hours. Ratios of less than 5 after 24 hours are good.