Hello, I have been stuck on this one for a few days, any advise would be appreciated:
I am doing a configuration of Serviceguard with Veritas Cluster File System on HP-UX 11i v3 and Oracle 10g R2 RAC. Right now I have Serviceguard running on 2 Integrity Virtual machines which are hosted on an rx8640 server.
<root@vguest1:/>swlist | grep -i serviceguard T1905CA A.11.19.00 Serviceguard T2777CB A.02.01 HP Serviceguard Cluster File System for RAC PHSS_40152 1.0 Serviceguard A.11.19.00
I loaded the latest Patches from the ITRC website for Serviceguard Storage Management Suite A.02.01 for HP-UX 11i v3.
The cluster starts and runs just fine: <root@vguest2:/etc/cmcluster/cfs>cmviewcl -v
CLUSTER STATUS cluster1 up
NODE STATUS STATE vguest1 up running
Cluster_Lock_LVM: VOLUME_GROUP PHYSICAL_VOLUME STATUS /dev/vglock /dev/dsk/c0t0d0 up
Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 0/0/1/0 lan0 STANDBY up 0/0/2/0 lan1
NODE STATUS STATE vguest2 up running
Cluster_Lock_LVM: VOLUME_GROUP PHYSICAL_VOLUME STATUS /dev/vglock /dev/dsk/c0t2d0 up
Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 0/0/2/0 lan0 STANDBY up 0/0/1/0 lan1
The problem comes when I create and start the package: <root@vguest2:/etc/cmcluster/cfs>cfscluster config CVM is now configured <root@vguest2:/etc/cmcluster/cfs>cfscluster start Starting CVM...
I attached the txt document of the error output from there. But basically the second node errors saying a crucial package failed, and then the second one goes down with the same error, and both nodes reboot.
Any ideas for what may be causing this, or what I can try to fix it? I will attach the SG-CFS-pkg.log in my next post (this only lets me attach one file it looks like).
Thanks you for your help, Dima
Note: If you are the author of this question and wish to assign points to any of the answers, please login first.For more information on assigning points ,click
here
Thanks for the quick reply Steven. I attached the output from the commands that create the cluster from the beginning. After those commands is where I try to create the package and the error occurs:
Create a 1-node cluster on the vguest2 host and attempt to build the SG-CFS-pkg. If it starts without failure, focus your attention on the other server that TOCs.
I have only seen CVM/CFS TOCs occur when CFS-related filesets are not properly patched. Check (swlist) the affected server for properly installed patches to the CFS (vx*) filesets. The filesets that affect CFS are:
Also insure it is using the same version of VxVM (4.0/5.0 etc) as vguest2.
I tried to create the cluster on vguest2 with just itself as the node and run the package, and it also crashed. It gave this error message:
^GMessage from syslogd@vguest2 at Wed Oct 28 10:34:05 2009 ... vguest2 cmcld[25315]: Reason: A crucial package failed Executing "/usr/sbin/cmrunpkg -v SG-CFS-pkg" Unable to retrieve package status for package SG-CFS-pkg cmrunpkg: Unable to start some package or package instances Error: Failed to start CVM
So it looks like CVM is failing to start, which is something new I see. I also tried to do the same thing but with vguest1, and it gave the same error. Both of these machines have the same software loaded on them, and same patch level. I suspect you may be right with the patches, but I have installed everything I can think of.
Also, I don't know if this means anything, but I keep seeing these error messages in the SG-CFS-pkg.log:
10/28/09 10:34:04 cmrunserv -r 5 SG-CFS-sgcvmd >> /etc/cmcluster/cfs/SG-CFS-pkg. log 2>&1 /etc/cmcluster/cfs/SG-CFS-sgcvmd.sh 10/28/09 10:34:04 List current imported disk groups: 10/28/09 10:34:04 SG configuration monitor started (poll_interval = 10) /etc/cmcluster/cfs/SG-CFS-vxfsckd.sh[53]: /usr/sbin/vxfsckd: not found. 10/28/09 10:34:04 ERROR: Unable to determine pid of vxfsckd
...and...
10/28/09 10:32:37 Starting VXFEN 10/28/09 10:32:42 Enabling cluster ODM 10/28/09 10:33:48 /sbin/init.d/odm start (exit=1) ERROR: The module 'odm' has a dependency on the module 'fdd' that cannot be satisfied.
Also, I think you were going to list the filesets that affect CFS in your post, but it didn't show up, could you please list those again.
If you don't see the file - it's not installed and likely neither is the fileset which CVM needs!
2) The messages you see about ODM dependency on fdd not being satisfied, reiterates the problem. Apparently not all required filesets have been installed.
As for the list of filesets that are supplied with T2775CB (HP Serviceguard Cluster File System), it is too long for this forum. If you would like to get it, please email me and I will send you the list. But since ODM has unsatisfied dependencies, it's very likely the SG product bundle did not install properly.
As for patches - please use this website to get a list and download currently recommended patches for the version of SMS (storage management suite) installed on your servers: http://www.hp.com/go/sgsms/patches
Thank you for your reply, that definitely makes sense now why they keep crashing. I am missing the /usr/sbin/vxfsckd file. ...and... <root@vguest2:/>swlist -l file | grep vxfsckd PHSS_40152.CM-CVM-CFS-COM: /etc/cmcluster/cfs/SG-CFS-vxfsckd.sh Package-CVM-CFS.CM-CVM-CFS-COM: /etc/cmcluster/cfs/SG-CFS-vxfsckd.sh
I do remember that certain components had errors when installing Serviceguard CFS for RAC. I am not exactly sure why thought. I am installing the newest version of the HP-UX OS I have, 11.31.0909. Previously I was working on 11.31.0709. I will reinstall Serviceguard, and let you know if that fixes the missing /usr/sbin/vxfsckd file. Is there a way to install only that component?
Dima, Your swlist indicates that there is a patch (PHSS_40152) installed on vxfsckd.sh (which is part of the VRTSvxfs fileset). You might be able to drill into the T2777CB product (in swinstall GUI) and reinstall just VRTSvxfs but you will also have to reinstall the patch too. I don't experiment with such things, so I just don't know for certain. I hope it goes well for you.
Ok, so I reinstalled one of the machines with 11.31.0909 and loaded the latest SG CFS for RAC, and installed the latest patches from the itrc website. I am still getting the package failed TOC, but it seems like something else it messing up. From the message below it says that the LLT failed to start, but when my system boots, the start-up progress shows that LLT is started. However, start-up progress shows that Starting VXFEN ......FAIL*. Before I installed some patches for VXFEN, the package syslog had an error message about vxfen failing, but after installing patches that went away (but it still fails on the start-up progress)...I am confused. Is there a patch for LLT that I also need to install outside the bundle?
Right now I am just trying to get this setup to work on one server (ie cluster with one node). I made sure that I downloaded the latest patches from the HP website, at this point the only thing I can think of trying is going back to the "recommended" or the older patches instead of the newest patches that I got from the website. Although I don't think that will fix my issue, I'm going to give that a shot.
Ok, FINALLY got it up and working. Here is what was wrong with my setup just in case anyone else runs into this issue:
I went back and checked the install of HP Serviceguard Cluster File System for RAC, and saw that not all of the components got installed, there were some errors during the install. After reading the error messages, the system was missing Base-VxTools-50 and SGeRAC (don't know why it needed it, is that normal???). After installing those two, I went back and reinstalled SGCFS for RAC, and it completed without errors. I also applied all the newest patches for SG, SGCFSRAC, VXVM. The main underlying issue was that VXFEN was failing on system startup. After the complete install and reboot it finally started.
Now, right now it is working on one node (but still huge progress), I will need to go and reinstall the second node to match it and then will try to get the two working together. I will close the thread as soon as I have both up and running. Thanks for all the help!
The patches definitely did the trick, what baffles me is why those patches weren't listed in the SG installation document. Guess they assume certain things about your system before the install.
Printable version
