{"id":346,"date":"2015-04-10T11:43:57","date_gmt":"2015-04-10T11:43:57","guid":{"rendered":"http:\/\/faq.graven-ict.nl\/wordpress\/?p=346"},"modified":"2018-06-22T15:39:26","modified_gmt":"2018-06-22T15:39:26","slug":"kapotte-disk-vervangen-hp-smart-array","status":"publish","type":"post","link":"https:\/\/wiki.graven-ict.nl\/wordpress\/kapotte-disk-vervangen-hp-smart-array\/","title":{"rendered":"Kapotte Disk vervangen HP smart array"},"content":{"rendered":"<p>Voor een klant hebben we een Bigdata cluster (hadoop) ingericht. De Hadoop filesystemen liggen op interne disken gekoppeld via een HP smart array. Normaal gesproken zou je de disken dan in een RAID opstellen willen hebben om disk uitval eenvoudig zonder actie op het OS op te vangen, echter wil de hadoop de disken zo direct mogelijk benaderen, het cluster heeft de data 3 keer op disk staan en meet zelf welke disken traag zijn. Bij deze klant werken veel verschillende beheerders dus is er wel voor gekozen om de disken in een aparte volume group te houden, om verwarring te voor komen.<\/p>\n<p>als er problemen gemeld worden door de monitoring of de applicatie eigenaar, zal je in de logging meldingen terug vinden die duiden op disk problemen.\u00a0bv dmesg<\/p><div id=\"grave-135489024\" class=\"grave-content grave-entity-placement\"><script async src=\"\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js?client=ca-pub-1970734611769428\" crossorigin=\"anonymous\"><\/script><ins class=\"adsbygoogle\" style=\"display:inline-block;width:728px;height:90px;\" \ndata-ad-client=\"ca-pub-1970734611769428\" \ndata-ad-slot=\"1820224830\"><\/ins> \n<script> \n(adsbygoogle = window.adsbygoogle || []).push({}); \n<\/script>\n<\/div>\n<pre>\r\nend_request: critical target error, dev sdh, sector 392\r\nsd 2:0:0:3: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE\r\nsd 2:0:0:3: [sdh] Sense Key : Hardware Error [current]\r\nsd 2:0:0:3: [sdh] Add. Sense: Logical unit failure\r\nsd 2:0:0:3: [sdh] CDB: Read(10): 28 00 00 00 01 80 00 00 08 00\r\nend_request: critical target error, dev sdh, sector 384\r\nsd 2:0:0:3: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE\r\nsd 2:0:0:3: [sdh] Sense Key : Hardware Error [current]\r\n<\/pre>\n<p>hier gaat het dus om \/dev\/sdh en omdat de disken dus als jbod zijn ingericht zal je ook problemen zien met de lvm commando&#8217;s. Controleer met pvs welk lv \/ filesystem gekoppeld gekoppeld zijn.<\/p>\n<pre>\r\n# pvs\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 2000364175360: Input\/output error\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 2000364232704: Input\/output error\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 0: Input\/output error\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 4096: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 0: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365289472: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365371392: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 4096: Input\/output error\r\nCouldn't find device with uuid 8HXHzG-WATa-PHj0-ycje-nzkR-DSYc-fC3goc.\r\nPV VG Fmt Attr PSize PFree\r\n\/dev\/mapper\/mpathap2 vg.root lvm2 a-- 19.86g 7.09g\r\n\/dev\/mapper\/mpathb vg.appl lvm2 a-- 60.00g 26.97g\r\n\/dev\/sde vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdf vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdg vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdi vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdj vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdk vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdl vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdm vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdn vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdo vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdp vg.hadoopdisk lvm2 a-- 1.82t 0\r\nunknown device vg.hadoopdisk lvm2 a-m 1.82t 0\r\n<\/pre>\n<p>dat is dus lv.hadoop3 we gaan nu het FS met op lv.hadoop3 umount. en deze uit de \/ect\/fstab uit #<br \/>\nen gaan proberen het FS te umounten. in dit geval hoeven we ons geen zorgen te maken over de data want binnen het cluster is alles dubbel.<\/p>\n<pre>\r\n# df -Ph |grep lv.hadoop3\r\n\/dev\/mapper\/vg.hadoopdisk-lv.hadoop3 1.8T 713G 1.1T 41% \/hadoop3\r\n# umount \/hadoop3\r\n<\/pre>\n<p>Probeer nu de lv te verwijderen.<\/p>\n<pre>\r\n# lvremove \/dev\/vg.hadoopdisk\/lv.hadoop3\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 2000364175360: Input\/output error\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 2000364232704: Input\/output error\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 0: Input\/output error\r\n\/dev\/vg.hadoopdisk\/lv.hadoop3: read failed after 0 of 4096 at 4096: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 0: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365289472: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365371392: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 4096: Input\/output error\r\nCouldn't find device with uuid 8HXHzG-WATa-PHj0-ycje-nzkR-DSYc-fC3goc.\r\nDo you really want to remove active logical volume lv.hadoop3? [y\/n]: y\r\nLogical volume \"lv.hadoop3\" successfully removed\r\n<\/pre>\n<p>De disk is missing dus kan je hem niet meer opgeven met vgreduce<\/p>\n<pre>\r\n# pvs\r\n\/dev\/sdh: read failed after 0 of 4096 at 0: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365289472: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365371392: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 4096: Input\/output error\r\nCouldn't find device with uuid 8HXHzG-WATa-PHj0-ycje-nzkR-DSYc-fC3goc.\r\nPV VG Fmt Attr PSize PFree\r\n\/dev\/mapper\/mpathap2 vg.root lvm2 a-- 19.86g 7.09g\r\n\/dev\/mapper\/mpathb vg.appl lvm2 a-- 60.00g 26.97g\r\n\/dev\/sde vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/ - \/\r\n\/dev\/sdo vg.hadoopdisk lvm2 a-- 1.82t 0\r\n\/dev\/sdp vg.hadoopdisk lvm2 a-- 1.82t 0\r\nunknown device vg.hadoopdisk lvm2 a-m 1.82t 1.82t\r\n<\/pre>\n<p>daarom vgreduce met de optie removemissing<\/p>\n<pre>\r\n[root@lsrv3121 ~]# vgreduce --removemissing vg.hadoopdisk\r\n\/dev\/sdh: read failed after 0 of 4096 at 0: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365289472: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 2000365371392: Input\/output error\r\n\/dev\/sdh: read failed after 0 of 4096 at 4096: Input\/output error\r\nCouldn't find device with uuid 8HXHzG-WATa-PHj0-ycje-nzkR-DSYc-fC3goc.\r\nWrote out consistent volume group vg.hadoopdisk\r\n<\/pre>\n<p>We kunnen de disk nu binnen de hpacucli opzoeken de drive kan op verschillende plekken op failed staan. binnen de tool zijn er logildrives en physical drives<br \/>\nik check altijd eerst de pd physical drives, als de drive echt stuk is kan het heel lang duren.<\/p>\n<pre>\r\n[root@lsrv3121 ~]# hpacucli\r\nHP Array Configuration Utility CLI 9.10.22.0\r\nDetecting Controllers...Done.\r\nType \"help\" for a list of supported commands.\r\nType \"exit\" to close the console.\r\n\r\n=> ctrl slot=1 pd all show status\r\nphysicaldrive 1E:1:1 (port 1E:box 1:bay 1, 2 TB): OK\r\nphysicaldrive 1E:1:2 (port 1E:box 1:bay 2, 2 TB): OK\r\nphysicaldrive 1E:1:3 (port 1E:box 1:bay 3, 2 TB): OK\r\nphysicaldrive 1E:1:4 (port 1E:box 1:bay 4, 2 TB): OK\r\nphysicaldrive 1E:1:5 (port 1E:box 1:bay 5, 2 TB): OK\r\nphysicaldrive 1E:1:6 (port 1E:box 1:bay 6, 2 TB): OK\r\nphysicaldrive 1E:1:7 (port 1E:box 1:bay 7, 2 TB): OK\r\nphysicaldrive 1E:1:8 (port 1E:box 1:bay 8, 2 TB): OK\r\nphysicaldrive 1E:1:9 (port 1E:box 1:bay 9, 2 TB): OK\r\nphysicaldrive 1E:1:10 (port 1E:box 1:bay 10, 2 TB): OK\r\nphysicaldrive 1E:1:11 (port 1E:box 1:bay 11, 2 TB): OK\r\nphysicaldrive 1E:1:12 (port 1E:box 1:bay 12, 2 TB): OK\r\n<\/pre>\n<p>physiek lijkt alles goed. dan nog de logical drives bekijken<\/p>\n<pre>\r\n=> ctrl slot=1 ld all show status\r\nlogicaldrive 1 (1.8 TB, RAID 0): OK\r\nlogicaldrive 2 (1.8 TB, RAID 0): OK\r\nlogicaldrive 3 (1.8 TB, RAID 0): OK\r\nlogicaldrive 4 (1.8 TB, RAID 0): Failed\r\nlogicaldrive 5 (1.8 TB, RAID 0): OK\r\nlogicaldrive 6 (1.8 TB, RAID 0): OK\r\nlogicaldrive 7 (1.8 TB, RAID 0): OK\r\nlogicaldrive 8 (1.8 TB, RAID 0): OK\r\nlogicaldrive 9 (1.8 TB, RAID 0): OK\r\nlogicaldrive 10 (1.8 TB, RAID 0): OK\r\nlogicaldrive 11 (1.8 TB, RAID 0): OK\r\nlogicaldrive 12 (1.8 TB, RAID 0): OK\r\n<\/pre>\n<p>logical drive 4 geeft aan dat die stuk is. die gaan die gaan we dus verwijderen.<\/p>\n<pre>\r\n=> ctrl slot=1 ld 4 delete\r\nWarning: Deleting an array can cause other array letters to become renamed.\r\nE.g. Deleting array A from arrays A,B,C will result in two remaining\r\narrays A,B ... not B,C\r\n\r\nWarning: Deleting the specified device(s) will result in data being lost.\r\nContinue? (y\/n) y\r\n=>\r\n<\/pre>\n<p>de drive is nu klaar om gewisseld te worden. laat de drive vervangen of zet er zelf een nieuwe in.\u00a0als de disk vervangen is door log je weer in op hpacucli en zoek de\u00a0unassigned disk,\u00a0deze disk kan je gebruiken om eens logische drive aan te maken<\/p>\n<pre>\r\n=> ctrl slot=1 pd all show\r\nphysicaldrive 1E:1:4 (port 1E:box 1:bay 4, SATA, 2 TB, OK)\r\n\r\n=> ctrl slot=1 create type=ld drives=1E:1:4 raid=0\r\nWarning: Creation of this logical drive has caused array letters to become\r\nrenamed.\r\n<\/pre>\n<p>de disk word nu automatische door linux op gedetecteerd.<br \/>\nzoek hem op met fdisk -l, voeg hem weer toe aan de volume groep of maak een partion aan en zet er een filesysteem op vergeet de fstab niet terug te zetten.<\/p>\n<pre>\r\n# fdisk -l |grep ^Disk |grep sd\r\n\r\n# pvcreate \/dev\/sdt\r\nPhysical volume \"\/dev\/sdt\" successfully created\r\n# vgextend vg.hadoopdisk \/dev\/sdt\r\nVolume group \"vg.hadoopdisk\" successfully extended\r\n# lvcreate -l 476924 -n lv.hadoop3 vg.hadoopdisk \/dev\/sdt\r\nLogical volume \"lv.hadoop3\" created\r\n# mkfs.ext4 \/dev\/vg.hadoopdisk\/lv.hadoop3\r\nmke2fs 1.41.12 (17-May-2010)\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Voor een klant hebben we een Bigdata cluster (hadoop) ingericht. De Hadoop filesystemen liggen op interne disken gekoppeld via een HP smart array. Normaal gesproken zou je de disken dan in een RAID opstellen willen hebben om disk uitval eenvoudig zonder actie op het OS op te vangen, echter wil de hadoop de disken zo ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/wiki.graven-ict.nl\/wordpress\/kapotte-disk-vervangen-hp-smart-array\/\" title=\"Meer lezen...\">Lees meer<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[],"class_list":["post-346","post","type-post","status-publish","format-standard","hentry","category-lvm-filesystemen-raid-disk"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/posts\/346","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/comments?post=346"}],"version-history":[{"count":4,"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/posts\/346\/revisions"}],"predecessor-version":[{"id":350,"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/posts\/346\/revisions\/350"}],"wp:attachment":[{"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/media?parent=346"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/categories?post=346"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wiki.graven-ict.nl\/wordpress\/wp-json\/wp\/v2\/tags?post=346"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}