How to expand GlusterFS replicated clusters by one server

This has come up several times in the last week. “I have 2n servers with 2 or 4 bricks each and I want to add 1 more server. How do I ensure the new server isn’t a replica of itself?”

This isn’t a simple thing to do. When you add bricks, replicas are added with each set of bricks in the replica count. For “replica 2”, each 2 new bricks added forms a replica pair. To prevent your two new bricks on the new server from being replicas of themselves, you’ll need to move an old brick to the new server. This is done with the replace-brick command.

So first we move server1:/data/brick2 to server3:/data/brick2

volname=myvol1  
from=server1:/data/brick2  
to=server3:/data/brick2  
gluster volume replace-brick $volname $from $to start

Monitor for completion

watch gluster volume replace-brick $volname $from $to status

Once it’s completed then commit the change

gluster volume replace-brick $volname $from $to commit

Check your data to ensure it’s all working right. If not, panic! Well, I suppose you could come join us in the IRC channel to help you figure out why, but it really should just work. First thing we’re going to tell you is to check the logs, so might as well do that too.

Ok, now that your data’s all moved, your volume is completely operational and all it’s circuits are functioning perfectly, you’re ready to add your two new bricks.

Save yourself some time and just format the brick store that’s mounted at server1:/data/brick2. You’ll have to wipe it and it’s xattrs anyway, so that’s much quicker.

gluster volume add-brick $volname server1:/data/brick2 server3:/data/brick1  
gluster volume rebalance $volname start

And you’re all set.