GlusterFS 3.6.1 Add replacement HDD to a glusterfs volume

First, make sure a new HDD replaced and initialized correctly. In my example it’s /dev/sdc.

Creating linux primary partition, formating as xfs and mounting to the same mount point as it was before:
parted -a optimal /dev/sdc mkpart primary 0% 100%
mkfs.xfs -i size=512 /dev/sdc1
mount /dev/sdc1 /export/sdc1
mkdir -p /export/sdc1/brick1

gluster volume info gfsvol – BrickN server1-gluster:/export/sdc1/brick1 should be listed here

Volume Name: gfsvol
Type: Replicate
Volume ID: def9e71b-e3cd-4d25-a604-e53aa717e744
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1-gluster:/export/sdc1/brick1
Brick2: server2-gluster:/export/sdc1/brick1

gluster volume heal gfsvol full
Another transaction is in progress for gfsvol. Please try again after sometime.

ls -ltr /var/log/glusterfs

cat etc-glusterfs-glusterd.vol.log

[2017-05-17 23:57:24.089817] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/6a4503614e8aae691f635297ed0dca19.socket failed (Invalid argument)
The message "I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick server1-gluster:/export/sdc1/brick1 has disconnected from glusterd." repeated 39 times between [2017-05-17 23:55:27.065833] and [2017-05-17 23:57:24.091004]
[2017-05-17 23:57:27.090603] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/6a4503614e8aae691f635297ed0dca19.socket failed (Invalid argument)
[2017-05-17 23:57:27.091934] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick server1-gluster:/export/sdc1/brick1 has disconnected from glusterd.
[2017-05-17 23:57:30.091171] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/6a4503614e8aae691f635297ed0dca19.socket failed (Invalid argument)

gluster volume heal gfsvol full
Commit failed on server2-gluster. Please check log file for details.

cat /var/log/glusterfs/bricks/export-sdc1-brick1.log

[2017-05-17 23:35:29.469536] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.6.1 (args: /usr/sbin/glusterfsd -s server1-gluster --volfile-id gfsvol.server1-gluster.export-sdc1-brick1 -p /var/lib/glusterd/vols/gfsvol/run/server1-gluster-export-sdc1-brick1.pid -S /var/run/6a4503614e8aae691f635297ed0dca19.socket --brick-name /export/sdc1/brick1 -l /var/log/glusterfs/bricks/export-sdc1-brick1.log --xlator-option *-posix.glusterd-uuid=c876c33a-3605-4c5c-b773-8d5e85655111 --brick-port 49153 --xlator-option gfsvol-server.listen-port=49153)
[2017-05-17 23:35:29.608846] I [graph.c:269:gf_add_cmdline_options] 0-gfsvol-server: adding option 'listen-port' for volume 'gfsvol-server' with value '49153'
[2017-05-17 23:35:29.608880] I [graph.c:269:gf_add_cmdline_options] 0-gfsvol-posix: adding option 'glusterd-uuid' for volume 'gfsvol-posix' with value 'c876c33a-3605-4c5c-b773-8d5e85655111'
[2017-05-17 23:35:29.640944] I [rpcsvc.c:2142:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2017-05-17 23:35:29.641025] W [options.c:898:xl_opt_validate] 0-gfsvol-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2017-05-17 23:35:29.656007] E [posix.c:5604:init] 0-gfsvol-posix: Extended attribute trusted.glusterfs.volume-id is absent
[2017-05-17 23:35:29.656026] E [xlator.c:425:xlator_init] 0-gfsvol-posix: Initialization of volume 'gfsvol-posix' failed, review your volfile again
[2017-05-17 23:35:29.656034] E [graph.c:322:glusterfs_graph_init] 0-gfsvol-posix: initializing translator failed
[2017-05-17 23:35:29.656041] E [graph.c:525:glusterfs_graph_activate] 0-graph: init failed
[2017-05-17 23:35:29.656750] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (0), shutting down


“[2017-05-17 23:35:29.656007] E [posix.c:5604:init] 0-gfsvol-posix: Extended attribute trusted.glusterfs.volume-id is absent” – here it is. Let’s add extended attribute.

grep volume-id /var/lib/glusterd/vols/gfsvol/info | cut -d= -f2 | sed 's/-//g'
def9e71be3cd4d25a604e53aa717e744

setfattr -n trusted.glusterfs.volume-id -v 0xdef9e71be3cd4d25a604e53aa717e744 /export/sdc1/brick1

service glusterd restart

Just to confirm:
getfattr -d -m . -e hex /export/sdc1/brick1

Check if it’s getting synced:
while [ 1 ]; do du -sh /export/sdc1/brick1; sleep 30; done

Start heal process again
gluster volume heal gfsvol full

gluster volume heal gfsvol info

Links: https://serverfault.com/questions/710220/how-do-you-add-a-replacement-hdd-to-a-glusterfs-volume
https://joejulian.name/blog/replacing-a-brick-on-glusterfs-340/