Using MySQL Enterprise Backup on InnoDB Cluster – Member Restore Use Cases

By | November 26, 2018

This is a quick blog demonstrating a couple backup Restore uses cases within a MySQL InnoDB Cluster 8.0 setup.  The backup used was completed in a previous blog (part 2 in this series) with the  MySQL Enterprise Backup 8.0 utility.  I’ll then use that backup to build an additional member to add to the cluster.

This blog is part 3 in a 3 part series on MySQL InnoDB Cluster 8.0.13’s feature introductions, associated scripting we can use, devops approached enhancements and general capabilities.

Adding & Restoring IC Members w/MEB

Back References to other MySQL Blogs

This blog will also make reference to my prior blog where I discuss Backups with Persisted Configurations.  This is a critically important blog to refer to as InnoDB Cluster 8.0 members will “persist” configurations just as that blog has described.  Also that backup scripts supports InnoDB Cluster members very well.

For clarity, I’ll repost the RESTORE scripting part of that backup script here:

Current Status & Membership of the Cluster

I’m working with the MySQL InnoDB Cluster that I built for my prior blog in this series. Refer to both for context, and for the full set of commands used to build it.  Some commands will be revisited.

However, here is a status output of the cluster as it is now.

Restore Use Case #1 – Rebuild a Current Member

Let’s stop member named gr128 , and pretend that the file system was corrupted and it crashed.  Therefore, that backup we took on member gr129 will now come in very handy!!

We will do the following to restore the database files and get it back into the clustered setup:

  1. Stop the MySQL instance on gr128 if its not already stopped/dead
  2. Verify the status of the changed cluster, entry should be "(MISSING)"
  3. Restore the backup to member gr128
  4. Start the MySQL instance on gr128
  5. Verify the updated status, should now be  "ONLINE"

Here is the output of steps above

Restore Use Case #2 – Build a New Member to be added

In this particular scenario, we’re interested in growing the size of our cluster and wanting to add an additional member.  Here are the details of the new instance I have ready.

This one was setup in the exact same way as all the other servers, hence using the repeatable method I mentioned in the first blog post of this series.

Also, the steps here will be reminiscent of our prior scenario. In this case, they will be the following:

  1. Stop the MySQL instance on gr130 if its not already stopped
  2. Restore the backup to member gr130
  3. Start the MySQL instance on gr130
  4. Add the new member gr130 to the cluster
  5. Verify the updated status

Lets see these steps in action.

Restore Use Case #3 – Rebuild a member that was removed with the “force” option

The instructions for managing MySQL InnoDB Cluster members are vast.  You might encounter sitatuations where you are trying to remove a member from the group:

  • Usually for situations that server is no longer needed, or wanted.
  • Could be for long outages due to maintenance on the hardware where losing failure tolerance isn’t desired

When that  -- cluster remove-instance command is issued, it really wants to ensure that member is up to date and in a healthy state. That’s a great way to have a member leave.

https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-working-with-cluster.html#remove-instances-from-innodb-cluster

You’ll notice though in the documentation link above, there is a --force  option you have available to you.  If the utility times out on trying to get the member into an up to date state, then --force  will stop trying after the timeout period is over, and it will ensure that the cluster’s metadata is properly removed regarding that member.

Later, you may realize you want to introduce a “forced out” member back in.  In that situation, you just refer to Use Case #2 Restore a New Member in order to solve the situation.

*** Below is a status output where  gr130  is no longer needed but the metadata still shows it is a member.  Use --force  to remove it from the clustered topology ***

 

Hope this helps with your InnoDB Cluster solutions!!!

 

One thought on “Using MySQL Enterprise Backup on InnoDB Cluster – Member Restore Use Cases

  1. Pingback: InnoDB-Cluster-Devops-mysqlsh-command-usage – Select All From MySQL

Comments are closed.