SQL Server Service Pack Upgrade failing on SQL Server 2008 R2 clustered instance

Dear Friends,

Few months back it was one of those Sunday afternoons when you really want to spend some time doing nothing more than sit and relax when my phone rang up. It was one of my junior colleagues with whom I worked years back. I realized he is bit anxious but hesitant to tell actual reason for reaching me. Anyway, after few minutes of initial talks he finally disclosed that he is in office and in one of his environments SQL Server service pack upgrade is failing on SQL Server 2008 R2 clustered instance.

Ok, he did mention that MSI/MSP checks are clean, Cluster failover worked perfectly, SQL resources found to be fine pre/post failover and failback and quite confident that everything is in place so patching should go fine.

Realizing that I cannot ask him to share his desktop to have a look into the respective server I requested him if it is possible for him to share Detail.txt file so that I could at-least have a look and understand. He agreed to it did sent file to me. When I looked into the file I found it contains cluster verification error (see below);

2013-08-25 12:50:58 Slp: Rule ‘Cluster_IsOnlineIfClustered’ detection result: Is Cluster Online Results = True; Is Cluster Verfication complete = False; Verfication Has Warnings = True; Verification Has Errors = True; on Machine XXXXXXX

Error: Action “Microsoft.SqlServer.Configuration.SetupExtension.ValidateFeatureSettingsAction” threw an exception during execution.

2013-08-25 12:51:24 Slp: Microsoft.SqlServer.Setup.Chainer.Workflow.ActionExecutionException: The cluster resource is not available. (Exception from HRESULT: 0x8007138E) —> System.Runtime.InteropServices.COMException (0x8007138E): The cluster resource is not available.

Question was; when all resources are online i.e. none of them in failed state then from where did this error message come from? To narrow down the issue, I asked my colleague if it is feasible to reboot the nodes one by one and then check if any resource fails to come online. As they were in change window he decided they are going to reboot individual nodes.

   

Once done; I got to know 3 disk group resources failed to come online, that were added recently. I thought before my colleague reaches storage team worth to check if they can be brought online. Yes, when tried they did come online. Time for a second reboot to find out if failure was intermittent one or persistent. No luck, second reboot also reported all 3 disk groups in failed state. Now, it was time to involve storage team which revealed there were issues with IO path and as a result on every reboot added disks were being removed resulting failures. This was finally fixed by storage team and patching went successful. My friend actually missed to check out cluster logs/event logs and of course SQL logs to get to the reason of failure.

1_SQL_Server_SP_Upgrade_failing_on_SQL_Server_2008_R2_clustered_instance

I believe prior patching we should check for if cluster validation check is clean, any suspicious messages in event logs and FIX them at first place. Not to forget rebooting individual nodes prior patching can prove to be very useful (as observed here) though I do not want to be generic as may not apply in all environments.

Leave a comment if you have similar experience and underlying reason.

 

Regards

Kanchan Bhattacharyya

Like us on FaceBook Follow us on Twitter | Join the fastest growing SQL Server group on FaceBook

Follow me on TwitterFollow me on FaceBook

   

About Kanchan Bhattacharyya

Kanchan is an astute IT professional, a seasoned SQL Database Administrator with 13+ years of industry experience. A calculated risk taker with deep technical knowledge and has gained proficiency in database consulting and solutions across different environments. Kanchan holds MBA degree in IT and an International Executive MBA in Project Management. He is very passionate about SQL Server and holds MCSE (Data Platform), MCSA – SQL 2012, MCITP – SQL 2008 certifications. Currently he is focusing on cloud and latest releases of SQL Server. When not working, Kanchan likes to spend his time reading on new technical developments specifically on SQL Server and other related technologies.

View all posts by Kanchan Bhattacharyya →

One Comment on “SQL Server Service Pack Upgrade failing on SQL Server 2008 R2 clustered instance”

  1. Thanks for sharing it very valuable in cases when one is stuck in pack upgrades in clustered environment..

Leave a Reply

Your email address will not be published.