OpenWRT and Bare Metal BackUps: Difference between revisions
| Line 6: | Line 6: | ||
The "Backing Up" or BackUp Process is the first part of cloning a system too. So why not just use something like an external drive bay to clone the disk / SSD to another drive? Because that involves removing the drive. The end objective is to be able to do a live, bare metal cloning process. | The "Backing Up" or BackUp Process is the first part of cloning a system too. So why not just use something like an external drive bay to clone the disk / SSD to another drive? Because that involves removing the drive. The end objective is to be able to do a live, bare metal cloning process. | ||
==== Assumptions ==== | ====Assumptions==== | ||
In syntax examples, devices are assumed to be in /dev. | In syntax examples, devices are assumed to be in /dev. | ||
| Line 37: | Line 37: | ||
Copy the Root Partition (second partition on a 'stock' / 'standard' OpenWRT x86 installation): | Copy the Root Partition (second partition on a 'stock' / 'standard' OpenWRT x86 installation): | ||
*Syntax: <code>dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img skip=511 conv=noerror bs=512 status=progress</code> | *Using DD; | ||
**Syntax: <code>dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img skip=511 conv=noerror bs=512 status=progress</code> | |||
**Working Example: <code>dd if=/dev/sda of=/run/media/root/NTFS/Root.img skip=511 conv=noerror bs=512 status=progress</code> | |||
*Using CP; | |||
**cp | |||
...and finally, issue this command, just to make sure everything is flushed from cache / memory: <code>sync</code> | |||
=== Restoring a BackUp (Image File(s)) === | |||
The reason it is done in separate stages is for demonstration purposes. And in a recovery situation, it might be that only one "Partition" needs to be recovers. Plus DD isn't very efficient when it comes to 'empty space'. Yes, there's that "sparse" option, but it still has to scan / read the entire drive. | |||
===Syntax Notes=== | |||
=== | ====DD==== | ||
*if: Input File (or Device) | *if: Input File (or Device) | ||
| Line 56: | Line 63: | ||
...and finally, DD seems to be a Zero Based utility, similar to how Disks (and other items like Intel SATA Ports, etc.) are numbered starting from Zero ( 0 ). So setting skip=511 will skip sectors 0 through 511, for a total of 512 sectors. | ...and finally, DD seems to be a Zero Based utility, similar to how Disks (and other items like Intel SATA Ports, etc.) are numbered starting from Zero ( 0 ). So setting skip=511 will skip sectors 0 through 511, for a total of 512 sectors. | ||
=== Overlay, OverlayFS, Union Mount, SquashFS, F2FS, and more... === | ==== CP ==== | ||
* a: The purpose with this switch in the backup process is to preserve permissions, but the A actually stands for "archive" | |||
...and finally, watch out for "Live" versions of Linux, like Rocky Linux, that have an "alias" setup for the cp command (which in the case of Rocky Linux is cp -i, where the -i=interactive). So what difference does that make? Well, when you type CP, you're actually typing CP -i. So if you type CP --force / -f, it won't work as anticipated and you'll end up having to answer yes to every 'overwrite a file situation' (or 'clobber' in the CP venacular). Why, because the -i switch always trumps the --force / f switch, so the command you're actually typing is CP -i --force / -f, and --force / -f loses to -i. And the final question of "Why would anyone do that with CP?" Well, it seems to be done in the name of "safety" (and hair pulling). And that's a great example of someone trying to think for you. It is now on this website's Top 3 List of the Most Bone Headed Stupid Things Ever. | |||
===Overlay, OverlayFS, Union Mount, SquashFS, F2FS, and more...=== | |||
Real quick, here's what's going on with a typical Router based installation of OpenWRT that is stored on Flash Memory Storage. | Real quick, here's what's going on with a typical Router based installation of OpenWRT that is stored on Flash Memory Storage. | ||
| Line 63: | Line 76: | ||
How is it done? Using the Overlay / Union Mount capabilities of Linux. Sorry, not here to explain that fully, hence the above links. | How is it done? Using the Overlay / Union Mount capabilities of Linux. Sorry, not here to explain that fully, hence the above links. | ||
=== Backing up the /overlay Directory === | ===Backing up the /overlay Directory=== | ||
Ideally (since LVM isn't typically available to do "Volume Shadow Copies"), one would boot to an alternative OS, use DD to clone the /overlay "Partition" / "Device". Easy with an x86_64 based install of OpenWRT, but also very possible with a Router based on Flash Memory Storage too. For the Router scenario, again ideally, if the /overlay "Partition" / "Device" is on an external USB Flash Drive (as it should be if a router has a USB 3.0 port), just pop it into a "Linux Box" (virtual or otherwise) for backing up. It can also be done "live" without much chance of something being modified during the cloning process. | Ideally (since LVM isn't typically available to do "Volume Shadow Copies"), one would boot to an alternative OS, use DD to clone the /overlay "Partition" / "Device". Easy with an x86_64 based install of OpenWRT, but also very possible with a Router based on Flash Memory Storage too. For the Router scenario, again ideally, if the /overlay "Partition" / "Device" is on an external USB Flash Drive (as it should be if a router has a USB 3.0 port), just pop it into a "Linux Box" (virtual or otherwise) for backing up. It can also be done "live" without much chance of something being modified during the cloning process. | ||
* Make a BackUp: While booted to an alternate OS (Rocky Linux, another OpenWRT install on a USB Flash Drive, etc.), in this example, booted to another OS, do... Wait, but first; | *Make a BackUp: While booted to an alternate OS (Rocky Linux, another OpenWRT install on a USB Flash Drive, etc.), in this example, booted to another OS, do... Wait, but first; | ||
** Locate where the /overlay Partition is; | **Locate where the /overlay Partition is; | ||
*** LSBLK will show the connection between the /overlay Directory and what it is "connected to" (hint for x86 systems, look in the /boot/grub/grub.cfg file) | ***LSBLK will show the connection between the /overlay Directory and what it is "connected to" (hint for x86 systems, look in the /boot/grub/grub.cfg file) | ||
**** For an OpenWRT install with /overlay on a USB Flash Drive: /overlay will be on /dev/sdXy (where X is a letter and y is a number, example: sda1) | ****For an OpenWRT install with /overlay on a USB Flash Drive: /overlay will be on /dev/sdXy (where X is a letter and y is a number, example: sda1) | ||
**** For an OpenWRT install with /overlay on a SATA Disk Drive / SSD: /overlay will be on a Loop Device, roughly equivalent to a VHD or VHDX File where a file represents a storage device | ****For an OpenWRT install with /overlay on a SATA Disk Drive / SSD: /overlay will be on a Loop Device, roughly equivalent to a VHD or VHDX File where a file represents a storage device | ||
***** Use this command: losetup -a (it will display something like this: /dev/loop0: (Some Numbers) (/sda2), offset 3014656 (/dev/loop0 is the device and /sda2 is the "file", actually a real drive)) | *****Use this command: losetup -a (it will display something like this: /dev/loop0: (Some Numbers) (/sda2), offset 3014656 (/dev/loop0 is the device and /sda2 is the "file", actually a real drive)) | ||
** "Create" a LOOP Device (For an x86 (used interchangeably with x86_64, yes, yes, don't say it)); | **"Create" a LOOP Device (For an x86 (used interchangeably with x86_64, yes, yes, don't say it)); | ||
*** Generic Syntax: losetup -o OffsetInBytes /dev/loopX /WhatEverPath/WhatEverFileName | ***Generic Syntax: losetup -o OffsetInBytes /dev/loopX /WhatEverPath/WhatEverFileName | ||
*** OpenWRT specific Syntax: losetup -o 3014656 -fP /WhatEverPath/WhatEverFileName (3014656* is the offset picked by the OpenWRT developers, losetup will pick the next available LOOP#) | ***OpenWRT specific Syntax: losetup -o 3014656 -fP /WhatEverPath/WhatEverFileName (3014656* is the offset picked by the OpenWRT developers, losetup will pick the next available LOOP#) | ||
*** OpenWRT Syntax Example: losetup -o 3014656 -fP /mnt/sda1/OpenWRT_BackUp.img | ***OpenWRT Syntax Example: losetup -o 3014656 -fP /mnt/sda1/OpenWRT_BackUp.img | ||
** Mount the LOOP Device (again for an x86 system) | **Mount the LOOP Device (again for an x86 system) | ||
*** Example: mount -t f2fs /dev/loop1 /mnt/loop1 (mkdir /mnt/loop1 before this command of course) | ***Example: mount -t f2fs /dev/loop1 /mnt/loop1 (mkdir /mnt/loop1 before this command of course) | ||
** ...and that's it for x86 devices. From there, just cp | **...and that's it for x86 devices. From there, just cp | ||
<nowiki>*</nowiki>If you haven't spotted it yet, but you've been asking, "Where is that file that OpenWRT mounts a Directory named /overlay as a LOOP Device?" Answer: There is no file. It is simply a sector on the Disk / SSD Drive to the end of it. And you might ask, "But won't it be overwritten?" Answer: Nope, because everything before it is considered ROM (as in no write), so it won't expand. And yes that means the entire disk drive (well, at least the OpenWRT partitions) is considered a File that breaks down into a ROM Section and an /overlay Section. | <nowiki>*</nowiki>If you haven't spotted it yet, but you've been asking, "Where is that file that OpenWRT mounts a Directory named /overlay as a LOOP Device?" Answer: There is no file. It is simply a sector on the Disk / SSD Drive to the end of it. And you might ask, "But won't it be overwritten?" Answer: Nope, because everything before it is considered ROM (as in no write), so it won't expand. And yes that means the entire disk drive (well, at least the OpenWRT partitions) is considered a File that breaks down into a ROM Section and an /overlay Section. | ||
| Line 86: | Line 99: | ||
mount -t f2fs /dev/loop1 /mnt/loop1 | mount -t f2fs /dev/loop1 /mnt/loop1 | ||
=== More on the LOSETUP Command === | ===More on the LOSETUP Command=== | ||
==== Create a "Device" to Mount, IE Associate a File with a Loop Device Node[edit | edit source] ==== | ====Create a "Device" to Mount, IE Associate a File with a Loop Device Node[edit | edit source]==== | ||
losetup -fP WhatEverImageName.img (To see mounted "Devices": losetup -a), (f = File Name to configure as a Loop Device, P = Scan Partition Size, a = Show All Loop Devices, b = sector size) | losetup -fP WhatEverImageName.img (To see mounted "Devices": losetup -a), (f = File Name to configure as a Loop Device, P = Scan Partition Size, a = Show All Loop Devices, b = sector size) | ||
| Line 95: | Line 108: | ||
Loop Device: <nowiki>https://en.wikipedia.org/wiki/Loop_device</nowiki> | Loop Device: <nowiki>https://en.wikipedia.org/wiki/Loop_device</nowiki> | ||
==== Mount the Image[edit | edit source] ==== | ====Mount the Image[edit | edit source]==== | ||
mount -t squashfs /dev/loop7 /mnt/squash | mount -t squashfs /dev/loop7 /mnt/squash | ||
Keep in mind this only works with IMG Files created from Partitions. If an entire disk drive is imaged, the above method won't work. However... | Keep in mind this only works with IMG Files created from Partitions. If an entire disk drive is imaged, the above method won't work. However... | ||
==== Mount an Image of an Entire Disk[edit | edit source] ==== | ====Mount an Image of an Entire Disk[edit | edit source]==== | ||
If an entire disk is imaged with DD, then a "map" of it can be viewed with CFDISK: cfdisk /dev/WhatEverLoopDevice (Example: cfdisk /dev/loop5) (Remember, use losetup -a to see "Devices") | If an entire disk is imaged with DD, then a "map" of it can be viewed with CFDISK: cfdisk /dev/WhatEverLoopDevice (Example: cfdisk /dev/loop5) (Remember, use losetup -a to see "Devices") | ||
| Line 107: | Line 120: | ||
With the above information garnered from CFDISK, then mount individual Partitions: mount -t WhatEverFileSystem (squashfs, ext4, ntfs-3g, etc.) /dev/WhatEverLoopDevice /mnt/WhatEverMountPoint: mount -t squashfs /dev/loop6p2 /mnt/Loop6/p2 (Also: losetup -d /dev/WhatEverLoopDevice will delete the loop device) | With the above information garnered from CFDISK, then mount individual Partitions: mount -t WhatEverFileSystem (squashfs, ext4, ntfs-3g, etc.) /dev/WhatEverLoopDevice /mnt/WhatEverMountPoint: mount -t squashfs /dev/loop6p2 /mnt/Loop6/p2 (Also: losetup -d /dev/WhatEverLoopDevice will delete the loop device) | ||
==== Note on: LOSETUP -d WhatEverLoopDeviceName[edit | edit source] ==== | ====Note on: LOSETUP -d WhatEverLoopDeviceName[edit | edit source]==== | ||
Bad news. The delete command doesn't always work. After issuing the command on a legitimate Loop Device, and with no errors, the Loop Device will persist. Checking it with LOSETUP -a shows it is still there. Accessing it with CFDISK or MOUNT still works. Oh, well. Windows solution time. Reboot. | Bad news. The delete command doesn't always work. After issuing the command on a legitimate Loop Device, and with no errors, the Loop Device will persist. Checking it with LOSETUP -a shows it is still there. Accessing it with CFDISK or MOUNT still works. Oh, well. Windows solution time. Reboot. | ||
<br /> | <br /> | ||
| Line 134: | Line 147: | ||
OpenWRT specifically mentions GRUB 2 in their documentation. But the way it works on an x86_64 EXT4 installation of OpenWRT would convince anyone that it is Legacy GRUB (except for the GNU GRUB version 2.02 heading at the top of the screen). The way it is configured via /etc/grub/grub.cfg would also convince anyone that Legacy GRUB was being used. So, the conclusion is that GRUB 2.02 is indeed being used, but since it is backwards compatible, it is in 'Legacy Mode'. | OpenWRT specifically mentions GRUB 2 in their documentation. But the way it works on an x86_64 EXT4 installation of OpenWRT would convince anyone that it is Legacy GRUB (except for the GNU GRUB version 2.02 heading at the top of the screen). The way it is configured via /etc/grub/grub.cfg would also convince anyone that Legacy GRUB was being used. So, the conclusion is that GRUB 2.02 is indeed being used, but since it is backwards compatible, it is in 'Legacy Mode'. | ||
=== Rant on Backing Up OpenWRT (Not just the configuration files, but an entire 'Drive' (Flash, SSD, etc.))[edit | edit source] === | ===Rant on Backing Up OpenWRT (Not just the configuration files, but an entire 'Drive' (Flash, SSD, etc.))[edit | edit source]=== | ||
How can the entire OpenWRT installation be backed up in a fashion similar to Acronis products, Clonezilla, etc.? | How can the entire OpenWRT installation be backed up in a fashion similar to Acronis products, Clonezilla, etc.? | ||
The first answer is that there is no built in facility to do that. Dang! Why not? Well, first of all, OpenWRT is intended to be installed on small router devices where this type of backup (Disk / Drive Imaging, Bare Metal BackUp, etc.) just isn't a thing. Second, and largely because of something mentioned in the first item, is that a program like that can't be crammed into a small enough space to fit on small router device. | The first answer is that there is no built in facility to do that. Dang! Why not? Well, first of all, OpenWRT is intended to be installed on small router devices where this type of backup (Disk / Drive Imaging, Bare Metal BackUp, etc.) just isn't a thing. Second, and largely because of something mentioned in the first item, is that a program like that can't be crammed into a small enough space to fit on small router device. | ||
==== Spoiler Alert: RESTIC[edit | edit source] ==== | ====Spoiler Alert: RESTIC[edit | edit source]==== | ||
What about Baquacil, err, wait, that's a pool chemical. Bactine! No, that's the antiseptic. RESTIC! Yup, that's the utility for backing up in OpenWRT (the only one) | What about Baquacil, err, wait, that's a pool chemical. Bactine! No, that's the antiseptic. RESTIC! Yup, that's the utility for backing up in OpenWRT (the only one) | ||
| Line 148: | Line 161: | ||
Wah! But what if I want to do it anyway because... Well, because I want to | Wah! But what if I want to do it anyway because... Well, because I want to | ||
==== Other Choices[edit | edit source] ==== | ====Other Choices[edit | edit source]==== | ||
Clonezilla? Yup, great, but one issue. One has to boot from the Clonezilla CD / DVD / USB Flash Drive / ISO to do it. Router off, Clonezilla on. | Clonezilla? Yup, great, but one issue. One has to boot from the Clonezilla CD / DVD / USB Flash Drive / ISO to do it. Router off, Clonezilla on. | ||
Again; WAH! I want to do it with the Router still fully functional, IE from within OpenWRT. | Again; WAH! I want to do it with the Router still fully functional, IE from within OpenWRT. | ||
==== What About DD?[edit | edit source] ==== | ====What About DD?[edit | edit source]==== | ||
Perfect! Except for one thing: Files might change right in the middle of a backup (because it takes a long time). Remember DD has no capability or concept of "Volume Shadow Copy" (Note: RESTIC doesn't have the capability, but it does have the concept of it, however that would require converting the entire OpenWRT install over to an LVM based Disk / SSD / Whatever 'Layout'.) | Perfect! Except for one thing: Files might change right in the middle of a backup (because it takes a long time). Remember DD has no capability or concept of "Volume Shadow Copy" (Note: RESTIC doesn't have the capability, but it does have the concept of it, however that would require converting the entire OpenWRT install over to an LVM based Disk / SSD / Whatever 'Layout'.) | ||
==== Ideal Objective[edit | edit source] ==== | ====Ideal Objective[edit | edit source]==== | ||
Ideally, the only solution would seem to be something like Acronis for Linux (or some other commercial product). Or something like RUBackUp and / or FSArchiver. The latter requiring the above noted conversion of OpenWRT over to an LVM based Disk / SSD / FlashDrive system. That's a lot of work. The best option would be making DD aware of Volume Snapshots and doing it that way. Ongoing work with this experiment will be to attempt changing the disk layout to include an LVM volume that allows for "Volume Shadow Copy" (Volume Snapshot, etc., IE, the Linux equivalent of that Windows feature) and using DD for a Bare Metal BackUp! | Ideally, the only solution would seem to be something like Acronis for Linux (or some other commercial product). Or something like RUBackUp and / or FSArchiver. The latter requiring the above noted conversion of OpenWRT over to an LVM based Disk / SSD / FlashDrive system. That's a lot of work. The best option would be making DD aware of Volume Snapshots and doing it that way. Ongoing work with this experiment will be to attempt changing the disk layout to include an LVM volume that allows for "Volume Shadow Copy" (Volume Snapshot, etc., IE, the Linux equivalent of that Windows feature) and using DD for a Bare Metal BackUp! | ||
==== Cop Out, err, Respect![edit | edit source] ==== | ====Cop Out, err, Respect![edit | edit source]==== | ||
Please remember, none of this is rant is intended to be disrespectful to the OpenWRT, RESTIC, or anyone else that might have felt slighted. It is simply a colorful way of noting how in the software world, once someone is given something, the inevitable question and desire of 'What else can it do?" comes up. That's what this is about | Please remember, none of this is rant is intended to be disrespectful to the OpenWRT, RESTIC, or anyone else that might have felt slighted. It is simply a colorful way of noting how in the software world, once someone is given something, the inevitable question and desire of 'What else can it do?" comes up. That's what this is about | ||