{"id":371,"date":"2017-05-10T16:34:26","date_gmt":"2017-05-10T16:34:26","guid":{"rendered":"http:\/\/www.micha.name\/blog\/?p=371"},"modified":"2017-05-10T18:08:09","modified_gmt":"2017-05-10T18:08:09","slug":"recovering-an-lvm-on-a-formatted-disk","status":"publish","type":"post","link":"https:\/\/www.micha.name\/blog\/2017\/05\/10\/recovering-an-lvm-on-a-formatted-disk\/","title":{"rendered":"Recovering an LVM on a formatted disk"},"content":{"rendered":"<p>TL;DR: Always have an up-to-date backup of your system, no matter how much you trust it.\u00a0This type of recovery is a poor second option..<\/p>\n<ol>\n<li>Use\u00a0<strong>testdisk<\/strong> to recover lvm backup metadata from the \/etc directory, otherwise <strong>hexedit<\/strong>\u00a0to scrape the information manually from the disk<\/li>\n<li>Use the <strong>pvcreate<\/strong> and\u00a0<strong>vgcfgrestore<\/strong>\u00a0commands to restore the lvm metadata<\/li>\n<li>Fix the MBR\/grub if it was your boot disk<\/li>\n<\/ol>\n<h1>Background<\/h1>\n<p>Last year I upgraded and cross-graded my home PC&#8217;s disks from the following configuration:<\/p>\n<ul>\n<li>1x128GB SSD (half Linux LVM, half Windows 7)<\/li>\n<li>1x256GB HDD (Linux LVM)<\/li>\n<li>1x512GB HDD (NTFS data drive)<\/li>\n<\/ul>\n<p>to the following:<\/p>\n<ul>\n<li>1x128GB SSD (Windows 10)<\/li>\n<li>1x256GB SSD (Linux LVM)<\/li>\n<li>1x512GB HDD (NTFS data drive)<\/li>\n<\/ul>\n<p>To do so, I moved the LVM partitions from the 128GB SSD and 256GB HDD to the new 256GB SSD (*), then reformatted and reinstalled Windows on the 128GB SSD (Windows being only used for Elite Dangerous, Linux really being the primary OS I use).<\/p>\n<p>(*) I can&#8217;t remember\u00a0the precise steps. Definitely at one stage the 256GB HDD contained the entire Linux VG.<\/p>\n<h1>Disaster<\/h1>\n<p>Fast forward to this year, and the 256GB SSD recently packed it in. Panick ensues.Read on..<\/p>\n<p>So naturally,\u00a0first steps first, try to\u00a0take a snapshot of the disk. Initially an <strong>rsync<\/strong> of the entire filesystem, then a <strong>dd<\/strong>, then a <strong>ddrescue<\/strong>. Unfortunately it looks like the SSD is too far gone;\u00a0only a few hundred MB into the copy the drive just goes offline with hard errors on the ATA link.<\/p>\n<p>Next steps; try to find the original HDD\u00a0amongst my pile of old drives. I didn&#8217;t hold out much hope for this as I had recently gathered them all and done secure wipes in preparation to selling\/freecycling\/dumping the disks.<\/p>\n<p>Sure enough, disk after disk turned out to be blank with either nothing on it, random garbage, or\u00a0remnants of other systems.<\/p>\n<h1>Hope<\/h1>\n<p>Finally I found a 256GB disc which was empty, but actually partitioned. Closer examination with a hex editor showed that it used to contain an LVM VG created by my lost machine..<\/p>\n<p>First steps: take a <strong>dd<\/strong> of the disk.\u00a0Taking absolutely no chances with this one!<\/p>\n<p>Second steps: take a copy of the disk image to work with.<\/p>\n<span class=\"collapseomatic \" id=\"id69d2a91551f1d\"  tabindex=\"0\" title=\"Fig 1. The disk was partitioned as follows:\"    >Fig 1. The disk was partitioned as follows:<\/span><div id=\"target-id69d2a91551f1d\" class=\"collapseomatic_content \">\n<blockquote>\n<pre><strong># fdisk -l \/data\/archives\/DriveBackups\/20170509_urutu\/20170509_disk1.img<\/strong>\r\n\r\nDisk \/data\/archives\/DriveBackups\/20170509_urutu\/20170509_disk1.img: 233.8 GiB, 251000193024 bytes, 490234752 sectors\r\nUnits: sectors of 1 * 512 = 512 bytes\r\nSector size (logical\/physical): 512 bytes \/ 512 bytes\r\nI\/O size (minimum\/optimal): 512 bytes \/ 512 bytes\r\nDisklabel type: dos\r\nDisk identifier: 0x03080307\r\n\r\nDevice Boot Start End Sectors Size Id Type\r\n\/data\/archives\/DriveBackups\/20170509_urutu\/20170509_disk1.img1 * 2048 490233855 490231808 233.8G ef EFI (FAT\r\n<\/pre>\n<\/blockquote>\n<\/div>\n<h1>Recovery<\/h1>\n<p>Firstly I found <a href=\"http:\/\/blog.adamsbros.org\/2009\/05\/30\/recover-lvm-volume-groups-and-logical-volumes-without-backups\/\">this article<\/a> for restoring the LVM on a disk which is no longer configured as a PV. This proved to be a very useful starting point!<\/p>\n<p>Initially I followed the suggestions and used <strong>hexedit<\/strong> to save the various lvm metadata dumps on the disk (hint:\u00a0use search), but after dumping half a dozen or so I got bored.<\/p>\n<span class=\"collapseomatic \" id=\"id69d2a91551ff8\"  tabindex=\"0\" title=\"Fig 2: LVM details found using hexedit\"    >Fig 2: LVM details found using hexedit<\/span><div id=\"target-id69d2a91551ff8\" class=\"collapseomatic_content \">\n<blockquote>\n<pre>001065F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00106600 76 67 31 20 7B 0A 69 64 20 3D 20 22 7A 63 73 41 vg1 {.id = \"zcsA\r\n00106610 4A 64 2D 57 62 6E 79 2D 75 6C 70 49 2D 6C 4C 4B Jd-Wbny-ulpI-lLK\r\n00106620 62 2D 4D 78 77 55 2D 52 61 71 4A 2D 69 72 57 6C b-MxwU-RaqJ-irWl\r\n00106630 59 68 22 0A 73 65 71 6E 6F 20 3D 20 31 32 0A 66 Yh\".seqno = 12.f\r\n00106640 6F 72 6D 61 74 20 3D 20 22 6C 76 6D 32 22 20 23 ormat = \"lvm2\" #\r\n00106650 20 69 6E 66 6F 72 6D 61 74 69 6F 6E 61 6C 0A 73 informational.s\r\n00106660 74 61 74 75 73 20 3D 20 5B 22 52 45 53 49 5A 45 tatus = [\"RESIZE\r\n00106670 41 42 4C 45 22 2C 20 22 52 45 41 44 22 2C 20 22 ABLE\", \"READ\", \"\r\n00106680 57 52 49 54 45 22 5D 0A 66 6C 61 67 73 20 3D 20 WRITE\"].flags =\r\n00106690 5B 5D 0A 65 78 74 65 6E 74 5F 73 69 7A 65 20 3D [].extent_size =\r\n001066A0 20 38 31 39 32 0A 6D 61 78 5F 6C 76 20 3D 20 32 8192.max_lv = 2\r\n001066B0 35 36 0A 6D 61 78 5F 70 76 20 3D 20 32 35 36 0A 56.max_pv = 256.\r\n001066C0 6D 65 74 61 64 61 74 61 5F 63 6F 70 69 65 73 20 metadata_copies\r\n001066D0 3D 20 30 0A 0A 70 68 79 73 69 63 61 6C 5F 76 6F = 0..physical_vo\r\n001066E0 6C 75 6D 65 73 20 7B 0A 0A 70 76 30 20 7B 0A 69 lumes {..pv0 {.i\r\n001066F0 64 20 3D 20 22 57 31 48 47 68 77 2D 32 6C 44 56 d = \"W1HGhw-2lDV\r\n00106700 2D 6C 4E 42 77 2D 36 69 42 76 2D 38 7A 57 67 2D -lNBw-6iBv-8zWg-\r\n--- 20170509_disk1.img --0x106600\/0x3A70C70000--------------------------<\/pre>\n<\/blockquote>\n<\/div>\n<p>&nbsp;<\/p>\n<p>I had already used <strong>testdisk<\/strong> on the disk and found a bunch of partitions &#8211; obviously not real ones as they are LVM logical volumes so <strong>testdisk<\/strong> can&#8217;t be used to restore them. But I re-ran <strong>testdisk<\/strong>\u00a0and used it to successfully dump out the <strong>\/etc\/lvm\/backup<\/strong> folder. Now I had access to the last valid configuration of the LVM volume group in the system.<\/p>\n<span class=\"collapseomatic \" id=\"id69d2a9155205c\"  tabindex=\"0\" title=\"Fig 3: Showing partitions as found by &lt;strong&gt;testdisk&lt;\/strong&gt;:\"    >Fig 3: Showing partitions as found by <strong>testdisk<\/strong>:<\/span><div id=\"target-id69d2a9155205c\" class=\"collapseomatic_content \">\n<blockquote>\n<pre>TestDisk 6.14, Data Recovery Utility, July 2013\r\nChristophe GRENIER &lt;grenier@cgsecurity.org&gt; http:\/\/www.cgsecurity.org\r\n\r\nDisk \/data\/archives\/DriveBackups\/20170509_urutu\/20170509_disk1.img - 251 GB \/ 23\r\n Partition Start End Size in sectors\r\n&gt; Linux 0 65 2 32 227 3 524288\r\n Linux Swap 32 227 4 555 14 35 8388608\r\n Linux 555 14 36 1207 195 12 10485760\r\n Linux 1207 195 13 3165 227 6 31457280\r\n Linux 3165 227 7 3818 152 46 10485760\r\n Linux 3818 152 47 4471 78 23 10485760 Linux 4471 78 24 5776 184 40 20971520\r\n Linux 5124 4 1 18178 46 44 209715200\r\n\r\n\r\nStructure: Ok. Use Up\/Down Arrow keys to select partition. Use Left\/Right Arrow keys to CHANGE partition characteristics:\r\n*=Primary bootable P=Primary L=Logical E=Extended D=Deleted\r\nKeys A: add partition, L: load backup, T: change type, P: list files,\r\n Enter: to continue\r\next4 blocksize=1024 Sparse superblock, 268 MB \/ 256 MiB<\/pre>\n<\/blockquote>\n<\/div>\n<p>&nbsp;<\/p>\n<span class=\"collapseomatic \" id=\"id69d2a91552162\"  tabindex=\"0\" title=\"Fig 4: Showing some of the contents of the \/etc directory in the root partition (3rd partition in the list above):\"    >Fig 4: Showing some of the contents of the \/etc directory in the root partition (3rd partition in the list above):<\/span><div id=\"target-id69d2a91552162\" class=\"collapseomatic_content \">\n<blockquote>\n<pre>TestDisk 6.14, Data Recovery Utility, July 2013\r\nChristophe GRENIER &lt;grenier@cgsecurity.org&gt; http:\/\/www.cgsecurity.org\r\n Linux 555 14 36 1207 195 12 10485760\r\nDirectory \/etc\r\n Previous\r\n&gt;-rw-r--r-- 0 0 1222 1-Aug-2014 23:32 fstab\r\n drwxr-xr-x 0 0 4096 24-Jul-2014 02:21 ld.so.conf.d\r\n -rw-r--r-- 0 0 11 5-May-2013 11:00 debian_version\r\n drwxr-xr-x 0 0 4096 29-Jul-2014 21:31 default\r\n drwxr-xr-x 0 0 4096 29-Jun-2014 12:28 profile.d\r\n -rw-r--r-- 0 0 35 6-May-2013 11:33 issue lrwxrwxrwx 0 0 21 20-Jul-2014 11:11 os-release\r\n drwxr-xr-x 0 0 4096 1-Aug-2014 22:09 cron.daily\r\n -rw-r--r-- 0 0 1863 5-Mar-2014 04:18 bash.bashrc\r\n drwxr-xr-x 0 0 4096 27-Apr-2014 13:58 apt\r\n -rw-r--r-- 0 0 956 5-Feb-2014 04:41 mke2fs.conf\r\n -rw-r--r-- 0 0 367 19-May-2014 10:50 bindresvport.blacklist\r\n drwxr-xr-x 0 0 4096 1-Aug-2014 22:09 pam.d -rw-r--r-- 0 0 552 26-Jul-2008 21:44 pam.conf\r\n Next\r\nUse Left arrow to go back, Right to change directory, h to hide deleted files\r\n q to quit, : to select the current file, a to select all files\r\n C to copy the selected files, c to copy the current file<\/pre>\n<\/blockquote>\n<\/div>\n<h2>Mounting the disk-image partition<\/h2>\n<p>The disk contained a single partition starting at offset 2048 of type EFI, which matched the physical volume information in the LVM metadata I had recovered.<\/p>\n<p>First I used <strong>cfdisk<\/strong> to change the partition type to Linux LVM (8e), although not sure that was entirely necessary, and then mounted the\u00a0partition using <strong>losetup<\/strong> and <strong>kpartx<\/strong>, which resulted in a device node for the partition in\u00a0<strong>\/dev\/mapper\/loop0p1<\/strong>. This device node will be used later with the lvm commands.<\/p>\n<blockquote>\n<pre>#\u00a0cfdisk disk.img<\/pre>\n<pre>#\u00a0losetup \/dev\/loop0 disk.img<\/pre>\n<pre>#\u00a0kpartx -a \/dev\/loop0<\/pre>\n<\/blockquote>\n<h2>Recovering the Volume Group<\/h2>\n<blockquote><p><strong>HINT: Use the -t (test) parameter to try out the commands first!<\/strong><\/blockquote>\n<p>Now it was a fairly simple procedure to recreate the volume group.<\/p>\n<p>Firstly, this is an excerpt of the data contained in the lvm backup, take a note of the volume group name, and the physical volume id.<br \/>\n<span class=\"collapseomatic \" id=\"id69d2a9155219a\"  tabindex=\"0\" title=\"Fig 5. LVM Text Format Volume Group\"    >Fig 5. LVM Text Format Volume Group<\/span><div id=\"target-id69d2a9155219a\" class=\"collapseomatic_content \">\n<blockquote>\n<pre># Generated by LVM2 version 2.02.106(2) (2014-04-10): Fri Aug 1 22:29:02 2014\r\n\r\ncontents = \"Text Format Volume Group\"\r\nversion = 1\r\n\r\ndescription = \"Created *after* executing 'vgreduce vg1 \/dev\/sda1'\"\r\n\r\ncreation_host = \"urutu\" # Linux urutu 3.14-1-amd64 #1 SMP Debian 3.14.12-1 (2014-07-11) x86_64\r\ncreation_time = 1406928542 # Fri Aug 1 22:29:02 2014\r\n\r\n<mark>vg1<\/mark> {\r\n id = \"zcsAJd-Wbny-ulpI-lLKb-MxwU-RaqJ-irWlYh\"\r\n seqno = 51\r\n format = \"lvm2\" # informational\r\n status = [\"RESIZEABLE\", \"READ\", \"WRITE\"]\r\n flags = []\r\n extent_size = 8192 # 4 Megabytes\r\n max_lv = 256\r\n max_pv = 256\r\n metadata_copies = 0\r\n\r\nphysical_volumes {\r\n\r\npv0 {\r\n id = \"<mark>W1HGhw-2lDV-lNBw-6iBv-8zWg-yLGZ-KXNUm8<\/mark>\"\r\n device = \"\/dev\/sdb1\" # Hint only\r\n\r\nstatus = [\"ALLOCATABLE\"]\r\n flags = []\r\n dev_size = 490231808 # 233.761 Gigabytes\r\n pe_start = 2048\r\n pe_count = 59842 # 233.758 Gigabytes\r\n }\r\n }\r\n\r\nlogical_volumes {\r\n\r\nboot {\r\n id = \"C6YCzW-itiQ-R7uo-xEZD-0J2B-XSWc-9BzET3\"\r\n status = [\"READ\", \"WRITE\", \"VISIBLE\"]\r\n flags = []\r\n creation_host = \"mint\"<\/pre>\n<\/blockquote>\n<\/div>\n<p>Take a note of the physical volume&#8217;s id (highlighted above), and plug it into the following command to re-create the physical volume.<\/p>\n<blockquote>\n<pre>#\u00a0pvcreate -ff -u <mark>W1HGhw-2lDV-lNBw-6iBv-8zWg-yLGZ-KXNUm8<\/mark>\u00a0--restorefile <strong>etc\/lvm\/backup\/vg1<\/strong>\u00a0\/<strong>dev\/mapper\/loop0p1<\/strong><\/pre>\n<\/blockquote>\n<p>Use\u00a0<strong>pvscan<\/strong> to verify the physical volume information. You can use <strong>pvdisplay<\/strong> to verify the ID matches before continuing! In my case it looked like this:<\/p>\n<blockquote>\n<pre># pvscan\r\n  PV \/dev\/sdc1   VG vg1             lvm2 [233.76 GiB \/ 89.51 GiB free]\r\n  Total: 1 [233.76 GiB] \/ in use: 1 [233.76 GiB] \/ in no VG: 0 [0   ]\r\n\r\n# pvdisplay \/dev\/mapper\/loop0p1\r\n  --- Physical volume ---\r\n  PV Name               \/dev\/mapper\/loop0p1\r\n  VG Name               \r\n  PV Size               233.76 GiB\r\n  Allocatable           NO\r\n  PE Size               0   \r\n  Total PE              0\r\n  Free PE               0\r\n  Allocated PE          0\r\n  PV UUID               <mark>W1HGhw-2lDV-lNBw-6iBv-8zWg-yLGZ-KXNUm8<\/mark><\/pre>\n<\/blockquote>\n<p>Now rewrite the\u00a0volume group information using the following command:<\/p>\n<blockquote>\n<pre>#\u00a0vgcfgrestore -f <strong>etc\/lvm\/backup\/vg1<\/strong>\u00a0-v <mark>vg1<\/mark><\/pre>\n<\/blockquote>\n<p>You can now verify your volume group exists using\u00a0<strong>lvdisplay<\/strong>, check the filesystems using\u00a0<strong>fsck<\/strong>,\u00a0and mount them as normal, for example:<\/p>\n<blockquote>\n<pre># mount \/dev\/mapper\/vg1-root \/mnt\/recovered<\/pre>\n<\/blockquote>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recovering an LVM volume group from a formatted drive.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[11,56,29],"tags":[12,30],"_links":{"self":[{"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/posts\/371"}],"collection":[{"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/comments?post=371"}],"version-history":[{"count":11,"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/posts\/371\/revisions"}],"predecessor-version":[{"id":382,"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/posts\/371\/revisions\/382"}],"wp:attachment":[{"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/media?parent=371"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/categories?post=371"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.micha.name\/blog\/wp-json\/wp\/v2\/tags?post=371"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}