Partial Givebacks during Storage Failovers in NetApp’s Clustered Data ONTAP

By on 06/26/2015.

tugofwar

If you’ve ever used clustered Data ONTAP and done storage failover tests, you may have noticed something strange when you attempted a giveback.

cluster::> storage failover giveback -ofnode node1
Info: Run the storage failover show-giveback command to check giveback status.

cluster::> storage failover show
                              Takeover 
Node           Partner        Possible State Description 
-------------- -------------- -------- -------------------------------------
node1          node2          true     Connected to node2. Waiting
                                       for cluster applications to come
                                       online on the local node.
node2          node1          true     Connected to node1, Partial giveback
2 entries were displayed.

cluster::> storage failover show
                              Takeover 
Node           Partner        Possible State Description 
-------------- -------------- -------- -------------------------------------
node1          node2          true     Connected to node2
node2          node1          true     Connected to node1, Giveback
                                       of one or more SFO aggregates failed
2 entries were displayed.

Partial Givebacks (No takebacks)

So, what happened?

To better understand that, we need to understand how storage failovers work in clustered Data ONTAP and how they differ from storage failovers in 7-Mode.

In 7-Mode, a cluster was simply a 2 node high-availability (HA) pair. If something happened on one node, the other node would take ownership of the failed node’s storage and IP addresses and you’d be running everything on one node. That had its limitations, of course – for one, you couldn’t really use both nodes in true active/active clustering. Each node was its own entity and required you to connect to a different IP address to get to dedicated volumes and aggregates. Failovers were known only as “cluster failovers” or CFO. Everything on a node would fail over at once.

Clustered Data ONTAP extends the notion of the 7-Mode HA pair and chains together multiple HA pairs for additional places for things like IP addresses to fail over to. Plus, you could move volumes to any node in the cluster (up to 24 nodes in NAS only, 8 nodes in SAN clusters) for capacity flexibility, storage tiering, performance, etc. You still have the notion of HA pairs, but now all of the nodes in the cluster act as a single namespace.

What’s different in clustered Data ONTAP is that there are now two concepts of failover: storage failover (SFO) and cluster failover (CFO).

Cluster failover involves the node’s root volume and aggregate, which stores the replicated database (RDB), logs and is known as “mroot.” These aggregates should only contain node root volumes for various reasons, including performance, cluster health and failover considerations. These failover last, but are first to giveback because they are required to allow the cluster to return to full health. If these fail to giveback, we don’t want data aggregates on a node that’s dead in the water!

Storage failover involves the actual data aggregates. These aggregates are most important to production environments, and thus, failover first when catastrophe strikes. They are also susceptible to partial givebacks.

CFO and SFO are known as “HA policies.”

You can see aggregate HA policy types from the command line:

cluster::> storage aggregate show -fields ha-policy 
aggregate        ha-policy 
---------------- --------- 
aggr0_root_node1 cfo 
aggr0_root_node2 cfo 
aggr1_node1      sfo 
aggr1_node2      sfo 
4 entries were displayed.

How do I tell what happened on giveback?

There are a couple of ways to do this.

First, there’s always the event logs:

cluster::> event log show 
Time                Node             Severity      Event
------------------- ---------------- ------------- ---------------------------
6/16/2015 20:43:58 node2 INFORMATIONAL sfo.PartialGiveback: Node is in partial giveback state.
6/16/2015 20:43:58 node2 INFORMATIONAL sfo.retry.autoGiveback: Automatic giveback of SFO aggregates will be retried after 5 minutes.
6/16/2015 20:43:58 node2 ERROR sfo.giveback.failed: Giveback of aggregate aggr1_node1 failed due to Giveback was vetoed..
6/16/2015 20:43:58 node2 ERROR sfo.sendhome.subsystemAbort: The giveback operation of 'aggr1_node1' was aborted by 'lock_manager'.
6/16/2015 20:43:58 node2 ERROR gb.sfo.veto.lmgr.nonCA.locks: Could not complete giveback because of non-CA locks on volume ntfs(1)@vserver:ddbd1ca2-2d48-11e4-ad0e-005056a63da1 SFO aggregate aggr1_node1.

In the above, the giveback was “vetoed” by “lock_manager.” If that doesn’t make a ton of sense to you, lock_manager is basically a file lock on a NAS share. I cover file locking in a blog post called Multiprotocol File Locking and You.

If you want to see more information about a specific error message, use the -messagename, -time and -instance:

cluster::> event log show -node node2 -messagename sfo.sendhome.subsystemAbort -time "6/16/2015 20:43:58" -instance
              Node: node2
         Sequence#: 1236961
              Time: 6/16/2015 20:43:58
          Severity: ERROR
            Source: cf_giveback
      Message Name: sfo.sendhome.subsystemAbort
             Event: sfo.sendhome.subsystemAbort: The giveback operation of 'aggr1_node1' was aborted by 'lock_manager'.
 Corrective Action: Check the syslog/EMS output for a subsystem-specific reason for aborting the giveback operation. The corrective action is subsystem-specific and is detailed in the corrective action portion of the message. Follow the corrective action specified by the subsystem and then reissue the 'giveback' command. If you cannot perform the corrective action, then use the 'override-vetoes' option in the 'giveback' command to force the giveback.
       Description: This message occurs when the specified subsystem aborts the Storage Failover giveback operation for the aggregate.

But, there’s a more straightforward way…

In the previous command output, you may have noticed this hint:

Info: Run the storage failover show-giveback command to check giveback status.

Well, it’s a good idea to use that:

cluster::> storage failover show-giveback 
               Partner
Node           Aggregate         Giveback Status
-------------- ----------------- ---------------------------------------------
node1
                                 No aggregates to give back
node2
               CFO Aggregates    Done
               aggr1_node1       Failed: Operation was vetoed by
                                 lock_manager. Giveback vetoed: Giveback
                                 cannot proceed because non-continuously
                                 available (non-CA) CIFS locks are present on
                                 the volume. Gracefully close the CIFS
                                 sessions over which non-CA locks are
                                 established. Use the "vserver cifs session
                                 file show -hosting-aggregate <aggregate
                                 list> -continuously-available No" command to
                                 view the open files that have CIFS sessions
                                 with non-CA locks established. <aggregate
                                 list> is the list of aggregates sent home as
                                 a result of the giveback operation. If lock
                                 state disruption for all existing non-CA
                                 locks is acceptable, retry the giveback
                                 operation by specifying "-override-vetoes
                                 true". Warning: Overriding vetoes to
                                 perform a giveback can be disruptive.
3 entries were displayed.

So in the above, we can see that CIFS/SMB is our culprit. To find the offending client:

cluster::> cifs session show
Node: node
Vserver: NAS
Connection Session                                   Open      Idle
ID         ID      Workstation      Windows User     Files     Time
---------- ------- ---------------- ---------------- --------- ---------------
369687504  1       10.228.225.120   DOMAIN\          3         44m 11s
                                    Administrator

Vetoed? What is this, Washington?

veto-bill

A “veto” simply means that the cluster, for your own data protection, has overruled a giveback of an aggregate for some reason. In the above, it was a file lock on a stateful NAS connection.

Other reasons for vetoes include:

  • Active snapmirrors
  • Failed disks
  • Unhealthy nodes
  • Coredump in progress
  • Volume moves in progress

In fact, you can get a list of these events from the CLI (you can do a lot from the CLI...):

cluster::> event route show -messagename gb.sfo*

So how do I fix it?

The answer to that question is a question: what’s broken and how important is it?

If it’s a open file, you are going to want to follow up with the client/user that has it open to see if they are really using it. The “idle time” portion of the “cifs session show” command gives a pretty good indication of if the file is in use.

Even if a file is in use, breaking a lock may not be harmful – generally, lock breaks only hurt things with active writes being committed.

If the veto is due to an active snapmirror, you can always resume it later. If it’s a failed disk, you probably should look into it, but with RAID-DP you are likely fine to proceed.

If in doubt, call NetApp support – they’ll help you sort it out.

If you do decide you want to ignore the veto, you can issue the equivalent “cf giveback -f” command with the option -override-vetoes. When you do it, you’ll get a nice scary warning:

cluster::> storage failover giveback -ofnode node1 -override-vetoes true 
Warning: Initiating a giveback with vetoes overridden will result in giveback proceeding even if the node detects outstanding issues that would make a giveback dangerous or disruptive. Do
 you want to continue? {y|n}: y
Info: Run the storage failover show-giveback command to check giveback status.

Some of this information is also covered in the NetApp KB What does partial giveback mean? This blog is essentially an update to that.

References

TR-4303: Logging in clustered Data ONTAP

Non-Disruptive Operations in clustered Data ONTAP

TR-4100: Non-Disruptive Operations and SMB File Shares for cDOT

TR-3982: Clustered Data ONTAP: An Introduction

Justin Parisi
Tech Mktg Engineer at NetApp
Justin is a Tech Marketing Engineer for all-things NFS around Data ONTAP at NetApp. He is a VMware vExpert, Cisco Champion, and a member of the NetApp A-Team. He also enjoys comic books, video games, photography, music, film, and current events/politics.

One Comment