Step1: Copy
Copy the entire contents of the RHEL 5.4 dvd to $rhroot, where $rhroot is any directory on your system.
Step 2: Create Kickstart
A kickstart file tells the installer how you want the system installed and allows your install to take place with no user input. In it, you configure options like the timezone, the partition layout, and which packages you want installed. The easiest way to create a kickstart config file is to use the one created when you installed your current system which is located at /root/anaconda-ks.cfg. See the documentation for more information on tweaking your kickstart file.
Step 3: Remove Unnecessary Packages, Add Extra Packages
Remove the rpms that you don't need from $rhroot/Server/. Most likely the Cluster, ClusterStorage, and VT directories of $rhroot can also be removed. The rpms you don't need are determined by what packages your installing as defined in the kickstart file. For any additional custom RPMs, be sure you add them to kickstart file or include them in a package group (see $rhroot/Server/comps-rhel5-server-core.xml).
Despite the fact that you may remove many RPMs, I haven't found any benefit to removing references to them from the package group file.
Step 4: Rebuild the Repository
Run the following command from $rhroot/Server to rebuild the Server repository to reflect any packages that were added or removed:
$ createrepo -u "media://`head -1 ../.discinfo`" -g repodata/comps-rhel5-server-core.xml .
Step 5: Build the ISO
From $rhroot run the following:
mkisofs -r -R -J -T -v -no-emul-boot -boot-load-size 4 -boot-info-table -b isolinux/isolinux.bin -c isolinux/boot.cat -x "lost+found" -o ../oel5Custom.iso .
Step 6: Test
I use VirtualBox to give my new ISO a spin.
Good References: