NOTE: This page was written in 2001 and may have invalid or old information, but for the most part, this info is still pretty valid.
What is MM5?
MM5 is a piece of software that takes a bunch of weather data and interpolates the data over time to make a forecast for a certain area. This forecast includes everything you can think of (moisture, air preassure, cloud cover, ...) and with the help of a package called Vis5D or VisAD you can manipulate this forecast in 3D.
For more information, see the NCAR MM5 Homepage.
What is MPI?
MPI stands for "Message-Passing Interface" and is a cross-platform method of inter-process communication. It's predecessor was PVM which stands for "Parallel Virtual Machine". Both of these APIs allow processes to communicate over a network in a heterogeneous environment without having to worry about byte-order or any of the other problems that often plague distributed hetergeneous programming.
For more information, see the Argonne National Labs' Computer Science Division's MPI page.
Why do this?
Max MIPS per $ - cheap computing power.
Today's desktop machines are growing in total CPU power much faster and can be purchased much cheaper than today's supercomputers. An 8-processor SGI Origin 2000 with 1GB of ram costs around $120K and the processors only come at 300Mhz. For the same ammount of money (May 1999 dollars) it is possible to set up 24 $5000 PC's which would have each 1GB of ram and 900-1100 Mhz of processing power each. This is a substantial performance/cost increase ($120K SGI = 2400Mhz, $120K Intel cluster = 24000Mhz - this is a factor of 10!).
How to set up the file system.
Since I had to have the same paths all across the system and I was unsure how the networking worked in the distributed model of MM5, I decided to go with NFS (I know, I know, ...). I'm storing all of the datasets on my personal machine and cross-mounting it on the remote nodes. The path is something like: /amnt/hostname/..../mm5mpi. Since the path is a "full" path to the actual data from any machine, this seems to work nicely, however because of the heavy NFS usage, the first half an hour of MM5 processing is moving data back and fourth which takes quite a while as compared to direct HD access.
Getting the software up and running.
This is the interesting part.
Get the following packages:
Un-tar them all somewhere in the exported directory structure and configure and build them accordingly:
- PGF77 - The Portland Group's F77 compiler.
- Pretty simple install. Just run the install script provided, add /usr/pgi/linux86/bin to your path and add the PGI environment variable to your .cshrc on your compile machine.
- NOTE: when licensing, it's machine-dependant, so be sure that you're not going to move this all to another machine or you'll need to buy a new license for the new machine.
- mpich - The public domain version of MPI.
- Make sure you specify the full path to the pgf77 compiler. It doesn't like to follow links for some reason.
- ./configure -prefix=/amnt/hostname/..../mpich -fc=/usr/pgi/linux86/bin/pgf77 -flinker=/usr/pgi/linux86/bin/pgf77
- make absolutely sure that the Fortran stuff is in there! IMPORTANT! Check the Makefile and if "NOF77 = 1" then configure didn't like your pgf77 path or something. Check the pgf77 compiler for correct licensing and stuff and re-run the mpich configure to re-probe for the pgf77 compiler. Using g77 for mpich and pgf77 for mm5 isn't a good idea and will not work.
- make
- make testing
- make install PREFIX=/amnt/hostname/..../mpich
- cd /amnt/hostname/..../mpich
- vi util/machines/machines.LINUX (add all of your hostnames - two entries if dual CPU)
- cd examples
- make all
- run some of the examples ("mpirun -arch LINUX -np 5 cpi")
- MM5 - MPP version
- extract both files (mm5v2.tar.Z and MPP.TAR.gz) in the same directory (MPP.TAR.gz last).
- alter your configure.user file to look like mine or just copy mine if you want (altering paths as appropriate). Note, the stuff in sections 4-7 should all reflect the appropriate info depending on the type of MM5 run you're going to execute. This stuff changes for each model. Ignore the configure.user.linux file.
- make mpp (not just make - that's a whole seperate beast)
- many things may not compile. If so, check out:
- MPP/RSL/Makefile.RSL:
- Add the path to the mpich includes in the INCLUDES section (-I/amnt/hostname/..../mpich/include and -I/amnt/hostname/..../mpich/build/LINUX/ch_p4/include)
- MPP/RSL/RSL/makefile.linux
- FC = pgf77 -byteswapio
- Add the path to the mpich includes in the CFLAGS section (-I/amnt/hostname/..../mpich/include and -I/amnt/hostname/..../mpich/build/LINUX/ch_p4/include)
- MPP/RSL/RSL/rsl_mpi_compat.c
- line 107, change "mpi_init__" to "mpi_init_" (remove the last underscore).
- If the final link is missing stuff, make sure that the RSL stuff has been made:
- cd MPP/RSL/RSL
- make linux
- Did everything link nicely?
- cd ../../..
- make mpp
- Run mm5.mpp
- cd Run
- ln -s /amnt/hostname/..../mpich/bin/mpirun
- ... then create the "mm5.deck" file as mentioned in your mm5 docs. You may need to generate one. Don't ask me what should go in this file, I just had Rong-Shyang Sheu make me up one
- Change the mm5.exe line near the bottom of the mm5.deck file to: "mpirun -arch LINUX -np 5 mm5.mpp"
- Get some data and run the mm5.deck file. You should be on your way!
- Drink a beer and get some sleep.
Our system specs.
Credits.
Thanks go to:
- John Michalakes - For porting MM5v2 to MPI and holding my hand during the initial compile.
- Rong-Shyang Sheu - For setting me up with the MM5 configuration.
- Laurie Carson - For working on porting the pre/post processing code to Linux.
- Scott Swerdlin - For allowing me to spend time on this project.
|