Starting with GlusterFS 3.3, one change has been the check to see if a directory (or any of it’s ancestors) is already part of a volume. This is causing many support questions in #gluster.
This was implemented because if you remove a brick from a volume and continue to use the volume, you can get file into a state where re-adding a former brick can cause all sort of problems, many of which can result in data loss.
If you’re going to reuse a brick, make sure you know what you’re doing.
For the directory (or any parent directories) that was formerly part of a volume, simply:
setfattr -x trusted.glusterfs.volume-id $brick_path setfattr -x trusted.gfid $brick_path rm -rf $brick_path/.glusterfs
Don’t worry if it says that the attribute does not exist. As long as it doesn’t exist, you’re in good shape.
Finally, restart glusterd to ensure it’s not “remembering” the old bricks.