OpenWRT and Bare Metal BackUps: Difference between revisions
| (5 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
The overall objective, long term objective of this project is to create a bare metal backup and move an x86_64 SquashFS based system to an LVM EXT4 based system. The "Backing Up" or BackUp Process or Cloning is of course the first part of that objective. | |||
For this article, the x86_64 (referred to as just x86 in this article) version of OpenWRT on an EXT 4 installed on a BIOS / MBR / DOS partitioned Disk / SSD will be the main focus. SquashFS will be addressed too. All of the below information was tested on an OpenWRT 19.07.10 EXT4 "Combined" (as in MBR, GRUB, Boot, and Root Partition (no Overlay)) based system. The origin and destination Boot Partition were set to 16MB in size and the origin and destination Root Partition were sized at 16GB. | |||
So why not just use something like an external drive bay to clone the disk / SSD to another drive? Because that involves removing the drive. The end objective is to be able to do a live, bare metal cloning process. Plus it doesn't do anything to accomplish migrating a SquashFS install to an EXT4 install on an LVM drive. | |||
===Backing Up / Cloning a Disk / SSD=== | |||
====Assumptions==== | ====Assumptions==== | ||
In syntax examples, devices are assumed to be in /dev. | In syntax examples, devices are assumed to be in /dev. | ||
==== | ====Components (Disk Sections) and Partitions that comprise an OpenWRT x86 Install==== | ||
*MBR (first Sector, 512 Bytes in size, Sector Number 0 (zero), contains stage 1.0 of GRUB and Partition table) | *MBR (first Sector, 512 Bytes in size, Sector Number 0 (zero), contains stage 1.0 of GRUB and Partition table) | ||
*"[[wikipedia:BIOS_boot_partition#Overview|MBR Gap]]" (contains stage 1.5 of GRUB and a file named core.img, but the image can't be seen because there is no file system there, it's just a 'gap') | *"[[wikipedia:BIOS_boot_partition#Overview|MBR Gap]]" (contains stage 1.5 of GRUB and a file named core.img, but the image can't be seen because there is no file system there, it's just a 'gap') | ||
**Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS. Even different versions of Windows have different size "Gaps". IE, the size is arbitrary and not set in stone. Lesson? Use a Partition Utility to look and see how things are arranged for an OS. It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap. | **Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS. Even different versions of Windows have different size "Gaps". IE, the size is arbitrary and not set in stone. Lesson? Use a Partition Utility to look and see how things are arranged for an OS. It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap. For OpenWRT, where the "MBR Gap" is about 256K, there's a LOT of unused space there. And no, didn't look to see exactly where in that space the CORE.IMG file. | ||
*Boot Partition (contains grub.cfg file with settings and VMLINUZ (a compressed Kernel Image in "Boot Executable bzImage" format) | *Boot Partition (contains grub.cfg file with settings and VMLINUZ (a compressed Kernel Image in "Boot Executable bzImage" format) | ||
*Root Partition (all the files that comprise an OpenWRT installation, plus user added files, packages, settings, etc. on an EXT4 installation) | *Root Partition (all the files that comprise an OpenWRT installation, plus user added files, packages, settings, etc. on an EXT4 installation) | ||
*Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations) | *Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations, offset exactly 3014656 Bytes from the beginning of the Root Partition, and addressed in more detail in a later section) | ||
====What to do / How to do it / Examples of | ====What to do / How to do it / Examples of Backing Up / Cloning an EXT4 Install of OpenWRT to an Image File==== | ||
Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress); | Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress); | ||
| Line 41: | Line 41: | ||
**Working Example: <code>dd if=/dev/sda of=/run/media/root/NTFS/Root.img skip=511 conv=noerror bs=512 status=progress</code> | **Working Example: <code>dd if=/dev/sda of=/run/media/root/NTFS/Root.img skip=511 conv=noerror bs=512 status=progress</code> | ||
*Using CP; | *Using CP; | ||
**cp | **cp -archive --force --verbose Source Destination | ||
**cp -a -f -v Source Destination | |||
**cp -afv Source Destination | |||
...and finally, issue this command, just to make sure everything is flushed from cache / memory: <code>sync</code> | ...and finally, issue this command, just to make sure everything is flushed from cache / memory: <code>sync</code> | ||
=== Restoring a | ===Restoring a Backed Up OpenWRT Installation from an Image File(s)=== | ||
The reason it is done in separate stages is for demonstration purposes. And in a recovery situation, it might be that only one "Partition" needs to be | The reason it is done in separate stages is for demonstration purposes. And in a recovery situation, it might be that only one "Partition" needs to be recovered, instead of an entire Disk / SSD. Plus DD isn't very efficient when it comes to 'empty space', so doing a full backup image of a Disk / SSD can consume a lot of space (unless zeros are written to the empty spaces on the Partition and the output of DD is piped into some form of compressed file. And yes, there's that "sparse" option for DD, but it still has to scan / read the entire drive, including all the empty space (because DD essentially does a Sector by Sector backup instead of a file system based backup) | ||
====Assumptions==== | |||
In syntax examples, if MBR and "MBR Gap" are cloned in one or two steps into one or two files (depending on user's choice), do it the same way on Restoring ("Assembly is the reverse of removal", in automotive repair manual vernacular). | |||
====What to do / How to do it / Examples of Restoring from an Image File==== | |||
Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress); | |||
*Syntax in one step; | |||
**<code>dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX bs=512</code> | |||
*Syntax in two steps (optional); | |||
**MBR: <code>dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX bs=512</code> | |||
**"MBR Gap": <code>dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX bs=512 seek=1</code> | |||
*Working Example (in one step): <code>dd if=/mnt/sda1/MBR_and_MBR-Gap.img of=/dev/sda conv=noerror bs=512 count=511 status=progress</code> | |||
**Note: Theoretically the count=511 doesn't need to be included in the above example because the source file should be the same size, IE 0-511. But if one is working from a "whole disk / SSD" backup Image, that will stop DD's writing at Sector 511, thus not overwriting the first, second, and probably other Partitions. | |||
Copy the Boot Partition (first partition on a 'stock' / 'standard' OpenWRT x86 installation): | |||
*Syntax: <code>dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX seek=512 conv=noerror bs=512 status=progress</code> | |||
*Working Example: <code>dd if=/run/media/root/NTFS/Boot.img of=/dev/sda seek=512 conv=noerror bs=512 status=progress</code> | |||
Copy the Root Partition (second partition on a 'stock' / 'standard' OpenWRT x86 installation): | |||
*...but first create the second Partition with CFDISK and format (mkfs.ext4 -L WhatEverLabel -v /dev/sdX ) it, if, that's ''if'' the second Partition doesn't already exist. | |||
*Using DD; | |||
**Syntax: <code>dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img of=/dev/sdX seek=WhatEverOffset conv=noerror bs=512 status=progress</code> | |||
**Working Example: <code>dd if=/dev/sda of=/run/media/root/NTFS/Root.img skip=511 of=/dev/sda seek=34816 conv=noerror bs=512 status=progress</code> | |||
*Using CP; | |||
**cp -archive --force --verbose Source Destination | |||
**cp -a -f -v Source Destination | |||
**cp -afv Source Destination | |||
**Note: If moving a SquashFS based installation of OpenWRT, one can copy the files from the existing SquashFS version of OpenWRT over a standard / initial / virgin installation of OpenWRT, thus 'converting' from SquashFS to EXT4 | |||
**Note 2: If doing the process in Note 1, keep in mind that there is also one thing to modify in the /boot/grub/grub.cfg file. Change this: rootfstype=squashfs To: rootfstype=ext4 | |||
...and finally, issue this command, just to make sure everything is flushed from cache / memory: <code>sync</code> | |||
===Syntax Notes=== | ===Syntax Notes=== | ||
| Line 63: | Line 101: | ||
...and finally, DD seems to be a Zero Based utility, similar to how Disks (and other items like Intel SATA Ports, etc.) are numbered starting from Zero ( 0 ). So setting skip=511 will skip sectors 0 through 511, for a total of 512 sectors. | ...and finally, DD seems to be a Zero Based utility, similar to how Disks (and other items like Intel SATA Ports, etc.) are numbered starting from Zero ( 0 ). So setting skip=511 will skip sectors 0 through 511, for a total of 512 sectors. | ||
==== CP ==== | ====CP==== | ||
* a: The purpose with this switch in the backup process is to preserve permissions, but the A actually stands for "archive" | *a: The purpose with this switch in the backup process is to preserve permissions, but the A actually stands for "archive" | ||
...and finally, watch out for "Live" versions of Linux, like Rocky Linux, that have an "alias" setup for the cp command (which in the case of Rocky Linux is cp -i, where the -i=interactive). So what difference does that make? Well, when you type CP, you're actually typing CP -i. So if you type CP --force / -f, it won't work as anticipated and you'll end up having to answer yes to every 'overwrite a file situation' (or 'clobber' in the CP venacular). Why, because the -i switch always trumps the --force / f switch, so the command you're actually typing is CP -i --force / -f, and --force / -f loses to -i. And the final question of "Why would anyone do that with CP?" Well, it seems to be done in the name of "safety" (and hair pulling). And that's a great example of someone trying to think for you. It is now on this website's Top 3 List of the Most Bone Headed Stupid Things Ever. | ...and finally, watch out for "Live" versions of Linux, like Rocky Linux, that have an "alias" setup for the cp command (which in the case of Rocky Linux is cp -i, where the -i=interactive). So what difference does that make? Well, when you type CP, you're actually typing CP -i. So if you type CP --force / -f, it won't work as anticipated and you'll end up having to answer yes to every 'overwrite a file situation' (or 'clobber' in the CP venacular). Why, because the -i switch always trumps the --force / f switch, so the command you're actually typing is CP -i --force / -f, and --force / -f loses to -i. And the final question of "Why would anyone do that with CP?" Well, it seems to be done in the name of "safety" (and hair pulling). And that's a great example of someone trying to think for you. It is now on this website's Top 3 List of the Most Bone Headed Stupid Things Ever. | ||
| Line 122: | Line 160: | ||
====Note on: LOSETUP -d WhatEverLoopDeviceName[edit | edit source]==== | ====Note on: LOSETUP -d WhatEverLoopDeviceName[edit | edit source]==== | ||
Bad news. The delete command doesn't always work. After issuing the command on a legitimate Loop Device, and with no errors, the Loop Device will persist. Checking it with LOSETUP -a shows it is still there. Accessing it with CFDISK or MOUNT still works. Oh, well. Windows solution time. Reboot. | Bad news. The delete command doesn't always work. After issuing the command on a legitimate Loop Device, and with no errors, the Loop Device will persist. Checking it with LOSETUP -a shows it is still there. Accessing it with CFDISK or MOUNT still works. Oh, well. Windows solution time. Reboot. | ||
====Example Uses for LOSETUP==== | |||
*Syntax: losetup -o OFFSET_IN_BYTES /dev/loopX /path/to/your/file | |||
*Example: losetup -o 3014656 -fP /mnt/sda1/OpenWRT_SDC2.img (this command allows for the LOSETUP command to select the next available LOOP Device in /dev) | |||
*Mount the resulting LOOP Device created by LOSETUP: mount -t f2fs /dev/loop1 /mnt/loop1 | |||
*Displays LOOP Devices: losetup -a | |||
*Delete LOOP Devices: losetup -d /dev/loopX | |||
===FAQs=== | ===FAQs=== | ||
| Line 131: | Line 177: | ||
*Why did the OpenWRT developers that focus so much on refinement and minimization use a whopping 256K for the MBR Gap? | *Why did the OpenWRT developers that focus so much on refinement and minimization use a whopping 256K for the MBR Gap? | ||
**The answer appears to be known only to them as there is no direct way to ask them. ChatGPT was asked and the answer was so bad, it was way worse than not knowing. | **The answer appears to be known only to them as there is no direct way to ask them. ChatGPT was asked and the answer was so bad, it was way worse than not knowing. | ||
*Example of saving to a Disk / SSD from an Image File: <code>dd if=/dev/sdb of=/mnt/sda1/MBR_and_MBR-Gap.img conv=noerror bs=512 count=511 status=progress</code> | *Example of saving to a Disk / SSD from an Image File: <code>dd if=/dev/sdb of=/mnt/sda1/MBR_and_MBR-Gap.img conv=noerror bs=512 count=511 status=progress</code> | ||
| Line 143: | Line 187: | ||
===Other Notes=== | ===Other Notes=== | ||
====LVM (Logical Volume Management) -- Rough Draft==== | |||
Signature ([[wikipedia:Master_boot_record#Disk_identity|Disk Signature / Disk Identity]]): A [[wikipedia:Master_boot_record#DISK_ID|32 bit section]] of the MBR (GPT, DOS, LVM, etc). Oh, and CFDISK won't work as it does not facilitate LVM Disks. | |||
*Steps for Creating a Bootable Disk with an LVM Partition | |||
**Create a BOOT Partition at an offset of 1M (equivalent to a quantity 2048 sectors that are 512 Bytes / Sector) | |||
**...same steps as below, but for /dev/sdaX | |||
*Steps for Creating an LVM Disk (NOTE: As in, that's an entire Disk, not just a Partition, and it assumes there's another disk to boot from, so do NOT do this, it is just here for reference) | |||
**PVCREATE (Create a Physical Disk, sort of equivalent to writing a Disk Signature): pvcreate /dev/sda (this will / can overwrite the DOS or GPT signature / identity of a disk) | |||
**VGCREATE (Create a Volume Group, nope, not sort of like creating a Partition (that's next), but rather like part B of PVCREATE, where one or more disks are assigned to a Volume Group): vgcreate VG1 /dev/sda (VG1 is an arbitrary name, it can be 'anything') | |||
**LVCREATE (Create a Logical Volume, equivalent to Partitioning); | |||
***lvcreate -L 16M -n BOOT VG1 (16M is the size of the first "Partition", BOOT is just a name given to the Logical Volume, it can be anything, VG1 is the name of the Volume Group from the previous step) | |||
***lvcreate -L 16G -n ROOT VG1 | |||
**Mount (...but Format first) the Volume Group(s) | |||
***LVDISPLAY (shows the available LVs in /dev/NameOfVolumeGroup/NameOfLogicalVolume) | |||
***mkfs.ext4 -L BOOT /dev/VG1/BOOT -v (where -L BOOT is just the name of the Partition) | |||
***mnt /dev/VG1/BOOT /mnt/BOOT (mkdir /mnt/BOOT first) | |||
*Steps for Creating an LVM "Partition" (IE, leave the MBR, "MBR Gap", and /boot Partitions intact. | |||
**...same as above, but instead of referring to an entire drive (/dev/sda), refer to a partition (/dev/sda1) | |||
====GRUB==== | ====GRUB==== | ||
OpenWRT specifically mentions GRUB 2 in their documentation. But the way it works on an x86_64 EXT4 installation of OpenWRT would convince anyone that it is Legacy GRUB (except for the GNU GRUB version 2.02 heading at the top of the screen). The way it is configured via /etc/grub/grub.cfg would also convince anyone that Legacy GRUB was being used. So, the conclusion is that GRUB 2.02 is indeed being used, but since it is backwards compatible, it is in 'Legacy Mode'. | OpenWRT specifically mentions GRUB 2 in their documentation. But the way it works on an x86_64 EXT4 installation of OpenWRT would convince anyone that it is Legacy GRUB (except for the GNU GRUB version 2.02 heading at the top of the screen). The way it is configured via /etc/grub/grub.cfg would also convince anyone that Legacy GRUB was being used. So, the conclusion is that GRUB 2.02 is indeed being used, but since it is backwards compatible, it is in 'Legacy Mode'. | ||
====QEMU ( QEMU-NDB )==== | |||
[[wikipedia:QEMU|QEMU]], specifically QEMU-NDB and QEMU-IMG, were commands available to install on OpenWRT that seemed like it would be useful in terms of VHD and other Virtual Disk Files, but were ultimately not needed. | |||
Example Command: qemu-nbd -c /dev/nbd0 /usr/src/vmlinuz/vmlinuz | |||
===Rant on Backing Up OpenWRT (Not just the configuration files, but an entire 'Drive' (Flash, SSD, etc.))[edit | edit source]=== | ===Rant on Backing Up OpenWRT (Not just the configuration files, but an entire 'Drive' (Flash, SSD, etc.))[edit | edit source]=== | ||