OpenWRT and Bare Metal BackUps: Difference between revisions

(2 intermediate revisions by the same user not shown)

Line 1:

The overall objective, long term objective of this project is to create a bare metal backup and move an x86_64 SquashFS based system to an LVM EXT4 based system. The "Backing Up" or BackUp Process or Cloning is of course the first part of that objective.

For this article, the x86_64 (referred to as just x86 in this article) version of OpenWRT on an EXT 4 installed on a BIOS / MBR / DOS partitioned Disk / SSD will be the main focus. SquashFS will be addressed too. All of the below information was tested on an OpenWRT 19.07.10 EXT4 "Combined" (as in MBR, GRUB, Boot, and Root Partition (no Overlay)) based system.

For this article, the x86_64 (referred to as just x86 in this article) version of OpenWRT on an EXT 4 installed on a BIOS / MBR / DOS partitioned Disk / SSD will be the main focus. SquashFS will be addressed too. All of the below information was tested on an OpenWRT 19.07.10 EXT4 "Combined" (as in MBR, GRUB, Boot, and Root Partition (no Overlay)) based system. The origin and destination Boot Partition were set to 16MB in size and the origin and destination Root Partition were sized at 16GB.

So why not just use something like an external drive bay to clone the disk / SSD to another drive? Because that involves removing the drive. The end objective is to be able to do a live, bare metal cloning process. Plus it doesn't do anything to accomplish migrating a SquashFS install to an EXT4 install on an LVM drive.

===Backing Up===

===Backing Up / Cloning a Disk / SSD===

====Assumptions====

In syntax examples, devices are assumed to be in /dev.

====~~Relevant "~~Partitions~~" of~~ an OpenWRT x86====

====Components (Disk Sections) and Partitions that comprise an OpenWRT x86 Install====

*MBR (first Sector, 512 Bytes in size, Sector Number 0 (zero), contains stage 1.0 of GRUB and Partition table)

*"[[wikipedia:BIOS_boot_partition#Overview|MBR Gap]]" (contains stage 1.5 of GRUB and a file named core.img, but the image can't be seen because there is no file system there, it's just a 'gap')

**Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS. Even different versions of Windows have different size "Gaps". IE, the size is arbitrary and not set in stone. Lesson? Use a Partition Utility to look and see how things are arranged for an OS. It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap.

**Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS. Even different versions of Windows have different size "Gaps". IE, the size is arbitrary and not set in stone. Lesson? Use a Partition Utility to look and see how things are arranged for an OS. It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap. For OpenWRT, where the "MBR Gap" is about 256K, there's a LOT of unused space there. And no, didn't look to see exactly where in that space the CORE.IMG file.

*Boot Partition (contains grub.cfg file with settings and VMLINUZ (a compressed Kernel Image in "Boot Executable bzImage" format)

*Root Partition (all the files that comprise an OpenWRT installation, plus user added files, packages, settings, etc. on an EXT4 installation)

*Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations)

*Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations, offset exactly 3014656 Bytes from the beginning of the Root Partition, and addressed in more detail in a later section)

====What to do / How to do it / Examples of Backing Up / Cloning to an Image File====

====What to do / How to do it / Examples of Backing Up / Cloning an EXT4 Install of OpenWRT to an Image File====

Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress);

Line 47:

...and finally, issue this command, just to make sure everything is flushed from cache / memory: <code>sync</code>

===Restoring a ~~BackUp~~ from an Image File(s)===

===Restoring a Backed Up OpenWRT Installation from an Image File(s)===

The reason it is done in separate stages is for demonstration purposes. And in a recovery situation, it might be that only one "Partition" needs to be ~~recovers~~. Plus DD isn't very efficient when it comes to 'empty space'. ~~Yes~~, there's that "sparse" option, but it still has to scan / read the entire drive.

The reason it is done in separate stages is for demonstration purposes. And in a recovery situation, it might be that only one "Partition" needs to be recovered, instead of an entire Disk / SSD. Plus DD isn't very efficient when it comes to 'empty space', so doing a full backup image of a Disk / SSD can consume a lot of space (unless zeros are written to the empty spaces on the Partition and the output of DD is piped into some form of compressed file. And yes, there's that "sparse" option for DD, but it still has to scan / read the entire drive, including all the empty space (because DD essentially does a Sector by Sector backup instead of a file system based backup)

====Assumptions====

In syntax examples, if MBR and "MBR Gap" are cloned in one or two steps, do it the same way on Restoring.

In syntax examples, if MBR and "MBR Gap" are cloned in one or two steps into one or two files (depending on user's choice), do it the same way on Restoring ("Assembly is the reverse of removal", in automotive repair manual vernacular).

====What to do / How to do it / Examples of Restoring from an Image File====

Line 63:

*Working Example (in one step): <code>dd if=/mnt/sda1/MBR_and_MBR-Gap.img of=/dev/sda conv=noerror bs=512 count=511 status=progress</code>

**Note: Theoretically the count=511 doesn't need to be ~~there~~ because the source file should be the same. But if one is working from a "whole disk / SSD" backup, that will stop DD's writing at Sector 511, thus not overwriting the first, second, and probably other Partitions.

**Note: Theoretically the count=511 doesn't need to be included in the above example because the source file should be the same size, IE 0-511. But if one is working from a "whole disk / SSD" backup Image, that will stop DD's writing at Sector 511, thus not overwriting the first, second, and probably other Partitions.

Copy the Boot Partition (first partition on a 'stock' / 'standard' OpenWRT x86 installation):

Line 187:

===Other Notes===

====LVM (Logical Volume Management) -- Rough Draft====

Signature ([[wikipedia:Master_boot_record#Disk_identity|Disk Signature / Disk Identity]]): A [[wikipedia:Master_boot_record#DISK_ID|32 bit section]] of the MBR (GPT, DOS, LVM, etc). Oh, and CFDISK won't work as it does not facilitate LVM Disks.

*Steps for Creating a Bootable Disk with an LVM Partition

**Create a BOOT Partition at an offset of 1M (equivalent to a quantity 2048 sectors that are 512 Bytes / Sector)

**...same steps as below, but for /dev/sdaX

*Steps for Creating an LVM Disk (NOTE: As in, that's an entire Disk, not just a Partition, and it assumes there's another disk to boot from, so do NOT do this, it is just here for reference)

**PVCREATE (Create a Physical Disk, sort of equivalent to writing a Disk Signature): pvcreate /dev/sda (this will / can overwrite the DOS or GPT signature / identity of a disk)

**VGCREATE (Create a Volume Group, nope, not sort of like creating a Partition (that's next), but rather like part B of PVCREATE, where one or more disks are assigned to a Volume Group): vgcreate VG1 /dev/sda (VG1 is an arbitrary name, it can be 'anything')

**LVCREATE (Create a Logical Volume, equivalent to Partitioning);

***lvcreate -L 16M -n BOOT VG1 (16M is the size of the first "Partition", BOOT is just a name given to the Logical Volume, it can be anything, VG1 is the name of the Volume Group from the previous step)

***lvcreate -L 16G -n ROOT VG1

**Mount (...but Format first) the Volume Group(s)

***LVDISPLAY (shows the available LVs in /dev/NameOfVolumeGroup/NameOfLogicalVolume)

***mkfs.ext4 -L BOOT /dev/VG1/BOOT -v (where -L BOOT is just the name of the Partition)

***mnt /dev/VG1/BOOT /mnt/BOOT (mkdir /mnt/BOOT first)

*Steps for Creating an LVM "Partition" (IE, leave the MBR, "MBR Gap", and /boot Partitions intact.

**...same as above, but instead of referring to an entire drive (/dev/sda), refer to a partition (/dev/sda1)

====GRUB====

@@ Line 1: / Line 1: @@
 The overall objective, long term objective of this project is to create a bare metal backup and move an x86_64 SquashFS based system to an LVM EXT4 based system.  The "Backing Up" or BackUp Process or Cloning is of course the first part of that objective.
-For this article, the x86_64 (referred to as just x86 in this article) version of OpenWRT on an EXT 4 installed on a BIOS / MBR / DOS partitioned Disk / SSD will be the main focus.  SquashFS will be addressed too.  All of the below information was tested on an OpenWRT 19.07.10 EXT4 "Combined" (as in  MBR, GRUB, Boot, and Root Partition (no Overlay)) based system.
+For this article, the x86_64 (referred to as just x86 in this article) version of OpenWRT on an EXT 4 installed on a BIOS / MBR / DOS partitioned Disk / SSD will be the main focus.  SquashFS will be addressed too.  All of the below information was tested on an OpenWRT 19.07.10 EXT4 "Combined" (as in  MBR, GRUB, Boot, and Root Partition (no Overlay)) based system.  The origin and destination Boot Partition were set to 16MB in size and the origin and destination Root Partition were sized at 16GB.
 So why not just use something like an external drive bay to clone the disk / SSD to another drive?  Because that involves removing the drive.  The end objective is to be able to do a live, bare metal cloning process.  Plus it doesn't do anything to accomplish migrating a SquashFS install to an EXT4 install on an LVM drive.
-===Backing Up===
+===Backing Up / Cloning a Disk / SSD===
 ====Assumptions====
 In syntax examples, devices are assumed to be in /dev.
-====Relevant "Partitions" of an OpenWRT x86====
+====Components (Disk Sections) and Partitions that comprise an OpenWRT x86 Install====
 *MBR (first Sector, 512 Bytes in size, Sector Number 0 (zero), contains stage 1.0 of GRUB and Partition table)
 *"[[wikipedia:BIOS_boot_partition#Overview|MBR Gap]]" (contains stage 1.5 of GRUB and a file named core.img, but the image can't be seen because there is no file system there, it's just a 'gap')
-**Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS.  Even different versions of Windows have different size "Gaps".  IE, the size is arbitrary and not set in stone.  Lesson?  Use a Partition Utility to look and see how things are arranged for an OS.  It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap.
+**Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS.  Even different versions of Windows have different size "Gaps".  IE, the size is arbitrary and not set in stone.  Lesson?  Use a Partition Utility to look and see how things are arranged for an OS.  It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap.  For OpenWRT, where the "MBR Gap" is about 256K, there's a LOT of unused space there.  And no, didn't look to see exactly where in that space the CORE.IMG file.
 *Boot Partition (contains grub.cfg file with settings and VMLINUZ (a compressed Kernel Image in "Boot Executable bzImage" format)
 *Root Partition (all the files that comprise an OpenWRT installation, plus user added files, packages, settings, etc. on an EXT4 installation)
-*Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations)
+*Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations, offset exactly 3014656 Bytes from the beginning of the Root Partition, and addressed in more detail in a later section)
-====What to do / How to do it / Examples of Backing Up / Cloning to an Image File====
+====What to do / How to do it / Examples of Backing Up / Cloning an EXT4 Install of OpenWRT to an Image File====
 Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress);
@@ Line 47: / Line 47: @@
 ...and finally, issue this command, just to make sure everything is flushed from cache / memory: <code>sync</code>
-===Restoring a BackUp from an Image File(s)===
+===Restoring a Backed Up OpenWRT Installation from an Image File(s)===
-The reason it is done in separate stages is for demonstration purposes.  And in a recovery situation, it might be that only one "Partition" needs to be recovers.  Plus DD isn't very efficient when it comes to 'empty space'.  Yes, there's that "sparse" option, but it still has to scan / read the entire drive.
+The reason it is done in separate stages is for demonstration purposes.  And in a recovery situation, it might be that only one "Partition" needs to be recovered, instead of an entire Disk / SSD.  Plus DD isn't very efficient when it comes to 'empty space', so doing a full backup image of a Disk / SSD can consume a lot of space (unless zeros are written to the empty spaces on the Partition and the output of DD is piped into some form of compressed file.  And yes, there's that "sparse" option for DD, but it still has to scan / read the entire drive, including all the empty space (because DD essentially does a Sector by Sector backup instead of a file system based backup)
 ====Assumptions====
-In syntax examples, if MBR and "MBR Gap" are cloned in one or two steps, do it the same way on Restoring.
+In syntax examples, if MBR and "MBR Gap" are cloned in one or two steps into one or two files (depending on user's choice), do it the same way on Restoring ("Assembly is the reverse of removal", in automotive repair manual vernacular).
 ====What to do / How to do it / Examples of Restoring from an Image File====
@@ Line 63: / Line 63: @@
 *Working Example (in one step): <code>dd if=/mnt/sda1/MBR_and_MBR-Gap.img of=/dev/sda conv=noerror bs=512 count=511 status=progress</code>
-**Note: Theoretically the count=511 doesn't need to be there because the source file should be the same.  But if one is working from a "whole disk / SSD" backup, that will stop DD's writing at Sector 511, thus not overwriting the first, second, and probably other Partitions.
+**Note: Theoretically the count=511 doesn't need to be included in the above example because the source file should be the same size, IE 0-511.  But if one is working from a "whole disk / SSD" backup Image, that will stop DD's writing at Sector 511, thus not overwriting the first, second, and probably other Partitions.
 Copy the Boot Partition (first partition on a 'stock' / 'standard' OpenWRT x86 installation):
@@ Line 187: / Line 187: @@
 ===Other Notes===
+====LVM (Logical Volume Management) -- Rough Draft====
+Signature ([[wikipedia:Master_boot_record#Disk_identity|Disk Signature / Disk Identity]]): A [[wikipedia:Master_boot_record#DISK_ID|32 bit section]] of the MBR (GPT, DOS, LVM, etc).  Oh, and CFDISK won't work as it does not facilitate LVM Disks.
+*Steps for Creating a Bootable Disk with an LVM Partition
+**Create a BOOT Partition at an offset of 1M (equivalent to a quantity 2048 sectors that are 512 Bytes / Sector)
+**...same steps as below, but for /dev/sdaX
+*Steps for Creating an LVM Disk (NOTE: As in, that's an entire Disk, not just a Partition, and it assumes there's another disk to boot from, so do NOT do this, it is just here for reference)
+**PVCREATE (Create a Physical Disk, sort of equivalent to writing a Disk Signature): pvcreate /dev/sda (this will / can overwrite the DOS or GPT signature / identity of a disk)
+**VGCREATE (Create a Volume Group, nope, not sort of like creating a Partition (that's next), but rather like part B of PVCREATE, where one or more disks are assigned to a Volume Group): vgcreate VG1 /dev/sda (VG1 is an arbitrary name, it can be 'anything')
+**LVCREATE (Create a Logical Volume, equivalent to Partitioning);
+***lvcreate -L 16M -n BOOT VG1 (16M is the size of the first "Partition", BOOT is just a name given to the Logical Volume, it can be anything, VG1 is the name of the Volume Group from the previous step)
+***lvcreate -L 16G -n ROOT VG1
+**Mount (...but Format first) the Volume Group(s)
+***LVDISPLAY (shows the available LVs in /dev/NameOfVolumeGroup/NameOfLogicalVolume)
+***mkfs.ext4 -L BOOT /dev/VG1/BOOT -v (where -L BOOT is just the name of the Partition)
+***mnt /dev/VG1/BOOT /mnt/BOOT (mkdir /mnt/BOOT first)
+*Steps for Creating an LVM "Partition" (IE, leave the MBR, "MBR Gap", and /boot Partitions intact.
+**...same as above, but instead of referring to an entire drive (/dev/sda), refer to a partition (/dev/sda1)
 ====GRUB====