{"id":696,"date":"2017-03-02T05:43:42","date_gmt":"2017-03-01T22:43:42","guid":{"rendered":"http:\/\/blog.trichev.com\/?p=696"},"modified":"2017-08-10T09:08:47","modified_gmt":"2017-08-10T02:08:47","slug":"glusterfs-3-6-1-split-brain-resolution","status":"publish","type":"post","link":"https:\/\/trichev.com\/blog\/2017\/03\/02\/glusterfs-3-6-1-split-brain-resolution\/","title":{"rendered":"GlusterFS 3.6.1 Split Brain resolution"},"content":{"rendered":"

Identify the bricks info:
\n[root@server1 ~]# gluster volume info images<\/code><\/p>\n

Volume Name: images
\nType: Replicate
\nVolume ID: e60b5d4b-be1f-4233-b09c-84a97001021f
\nStatus: Started
\nNumber of Bricks: 1 x 2 = 2
\nTransport-type: tcp
\nBricks:
\nBrick1: server1-gluster:\/export\/images\/brick1
\nBrick2: server2-gluster:\/export\/images\/brick1
\nOptions Reconfigured:
\ndiagnostics.count-fop-hits: on
\ndiagnostics.latency-measurement: on
\nperformance.quick-read: off
\nperformance.read-ahead: off
\nperformance.io-cache: off
\nperformance.stat-prefetch: off
\ncluster.eager-lock: enable
\nnetwork.remote-dio: enable
\ncluster.quorum-type: fixed
\ncluster.quorum-count: 1
\nstorage.owner-uid: 107
\nstorage.owner-gid: 107<\/code><\/p>\n

Get the path of the file that is in split-brain:
\nIt can be obtained either by
\na) The command gluster volume heal <volname> info split-brain<\/code>.<\/p>\n

[root@server1 ~]# gluster volume heal images info split-brain
\nGathering list of split brain entries on volume images has been successful<\/code><\/p>\n

Brick server1-gluster:\/export\/images\/brick1
\nNumber of entries: 1024
\nat path on brick
\n-----------------------------------
\n2017-02-08 06:14:56 \/srvmsim01v.img
\n2017-02-08 07:55:53 \/srvmmdb02v.img
\n2017-02-08 07:55:53 \/srvmmgw02v.img
\netc.<\/code><\/p>\n

Brick server2-gluster:\/export\/images\/brick1
\nNumber of entries: 1024
\nat path on brick
\n-----------------------------------
\n2017-03-01 09:36:44 \/srvmmgw02v.img
\n2017-03-01 09:37:45 \/srvmsim01v.img
\n2017-03-01 09:37:45 \/srvmmdb02v.img
\netc.<\/code><\/p>\n

b) Identify the files for which file operations performed from the client keep failing with Input\/Output error.<\/p>\n

Close the applications that opened this file from the mount point. In case of VMs, they need to be powered-off.<\/p>\n

Decide on the correct copy:
\nThis is done by observing the afr changelog extended attributes of the file on the bricks using the getfattr command; then identifying the type of split-brain (data split-brain, metadata split-brain, entry split-brain or split-brain due to gfid-mismatch); and finally determining which of the bricks contains the ‘good copy’ of the file.
\ngetfattr -d -m . -e hex <file-path-on-brick><\/code>.<\/p>\n

0x 000003d7 00000001 00000000\r\n\u00a0\u00a0\u00a0   |\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0 | \u00a0 \u00a0 \u00a0\u00a0 |\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 |\u00a0\u00a0\u00a0\u00a0 \u00a0 \u00a0|\u00a0\u00a0\u00a0\u00a0 \u00a0 \u00a0 \\_ changelog of directory entries\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 |\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0  \\_ changelog of metadata\r\n\u00a0\u00a0\u00a0\u00a0\u00a0  \\ _ changelog of data<\/pre>\n
It is also possible that one brick might contain the correct data while the other might contain the correct metadata.<\/p>\n
[root@server1 ~]# stat \/export\/images\/brick1\/srvmsim01v.img
\nFile: `\/export\/images\/brick1\/pny2msim01v.img'
\nSize: 15228796928 Blocks: 29743528 IO Block: 4096 regular file
\nDevice: 812h\/2066d Inode: 116 Links: 2
\nAccess: (0600\/-rw-------) Uid: ( 0\/ root) Gid: ( 0\/ root)
\nAccess: 2017-03-01 09:46:23.599184357 -0500
\nModify: 2017-03-01 12:23:04.693187960 -0500
\nChange: 2017-03-01 12:23:21.924583877 -0500<\/code><\/p>\n
[root@server1 ~]# md5sum \/export\/images\/brick1\/srvmsim01v.img
\ncb21a48ee44309cd0a2bcf6bec4c0f7c \/export\/images\/brick1\/srvmsim01v.img<\/code><\/p>\n
[root@server2 ~]# stat \/export\/images\/brick1\/srvmsim01v.img
\nFile: `\/export\/images\/brick1\/pny2msim01v.img'
\nSize: 15228796928 Blocks: 22730808 IO Block: 4096 regular file
\nDevice: 812h\/2066d Inode: 115 Links: 2
\nAccess: (0600\/-rw-------) Uid: ( 0\/ root) Gid: ( 0\/ root)
\nAccess: 2015-09-16 19:01:35.737767450 -0400
\nModify: 2017-03-01 12:23:17.320088786 -0500
\nChange: 2017-03-01 12:23:34.542382249 -0500<\/code><\/p>\n
[root@server2 ~]# md5sum \/export\/images\/brick1\/srvmsim01v.img
\n5062e0f3ef1a0a2c36825cd769366276 \/export\/images\/brick1\/srvmsim01v.img<\/code><\/p>\n
[root@server1 ~]# getfattr -d -m . -e hex \/export\/images\/brick1\/srvmsim01v.img
\ngetfattr: Removing leading '\/' from absolute path names
\n# file: export\/images\/brick1\/srvmsim01v.img
\ntrusted.afr.dirty=0x000000000000000000000000
\ntrusted.afr.images-client-0=0x000000000000000000000000
\ntrusted.afr.images-client-1=0x015457d20000000000000000
\ntrusted.gfid=0x43304ae0fa284e178e8364b837b30925<\/code><\/p>\n
[root@server2 ~]# getfattr -d -m . -e hex \/export\/images\/brick1\/srvmsim01v.img
\ngetfattr: Removing leading '\/' from absolute path names
\n# file: export\/images\/brick1\/srvmsim01v.img
\ntrusted.afr.dirty=0x000000000000000000000000
\ntrusted.afr.images-client-0=0x000000040000000000000000
\ntrusted.afr.images-client-1=0x000000000000000000000000
\ntrusted.gfid=0x43304ae0fa284e178e8364b837b30925<\/code><\/p>\n
So, different md5sum output, blocks, inodes, but the same size. Meta data is not corrupted. I decided to keep the first replica.<\/p>\n
Reset the relevant extended attribute on the brick(s) that contains the ‘bad copy’ of the file data\/metadata using the setfattr command.
\nsetfattr -n <attribute-name> -v <attribute-value> <file-path-on-brick><\/code><\/p>\n
[root@server2 ~]#setfattr -n trusted.afr.images-client-0 -v 0x000000000000000000000000 \/export\/images\/brick1\/srvmsim01v.img<\/code><\/p>\n
Trigger self-heal on the file by performing lookup from the client:
\nls -l <file-path-on-gluster-mount><\/code><\/p>\n
[root@server1 ~]# ls -l \/export\/images\/brick1\/srvmsim01v.img
\n-rw------- 2 qemu qemu 15236399104 Mar 1 17:37 \/export\/images\/brick1\/srvmsim01v.img
\n[root@server1 ~]# ls -l \/var\/lib\/libvirt\/images\/pny2msim01v.img
\n-rw------- 1 qemu qemu 15236399104 Mar 1 17:38 \/var\/lib\/libvirt\/images\/srvmsim01v.img<\/code><\/p>\n
Links:
\nhttps:\/\/gluster.readthedocs.io\/en\/latest\/Troubleshooting\/split-brain\/<\/a>
\nhttps:\/\/gluster.readthedocs.io\/en\/latest\/Troubleshooting\/heal-info-and-split-brain-resolution\/ (If you’re under 3.7 or higher))<\/a>
\nhttps:\/\/github.com\/gluster\/glusterfs\/blob\/master\/doc\/debugging\/split-brain.md<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Identify the bricks info: [root@server1 ~]# gluster volume info images Volume Name: images Type: Replicate Volume ID: e60b5d4b-be1f-4233-b09c-84a97001021f Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: server1-gluster:\/export\/images\/brick1 Brick2: server2-gluster:\/export\/images\/brick1 Options Reconfigured: diagnostics.count-fop-hits: on diagnostics.latency-measurement: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: fixed […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[231],"tags":[212,220,32,28,14,11],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/posts\/696"}],"collection":[{"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/comments?post=696"}],"version-history":[{"count":9,"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/posts\/696\/revisions"}],"predecessor-version":[{"id":706,"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/posts\/696\/revisions\/706"}],"wp:attachment":[{"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/media?parent=696"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/categories?post=696"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trichev.com\/blog\/wp-json\/wp\/v2\/tags?post=696"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}