OpenWRT and Bare Metal BackUps

The overall objective, long term objective of this project is to create a bare metal backup and move an x86_64 SquashFS based system to an LVM EXT4 based system. The "Backing Up" or BackUp Process or Cloning is of course the first part of that objective.

For this article, the x86_64 (referred to as just x86 in this article) version of OpenWRT on an EXT 4 installed on a BIOS / MBR / DOS partitioned Disk / SSD will be the main focus. SquashFS will be addressed too. All of the below information was tested on an OpenWRT 19.07.10 EXT4 "Combined" (as in MBR, GRUB, Boot, and Root Partition (no Overlay)) based system. The origin and destination Boot Partition were set to 16MB in size and the origin and destination Root Partition were sized at 16GB.

So why not just use something like an external drive bay to clone the disk / SSD to another drive? Because that involves removing the drive. The end objective is to be able to do a live, bare metal cloning process. Plus it doesn't do anything to accomplish migrating a SquashFS install to an EXT4 install on an LVM drive.

Backing Up / Cloning a Disk / SSD

Assumptions

In syntax examples, devices are assumed to be in /dev.

Components (Disk Sections) and Partitions that comprise an OpenWRT x86 Install

• MBR (first Sector, 512 Bytes in size, Sector Number 0 (zero), contains stage 1.0 of GRUB and Partition table)
• "MBR Gap" (contains stage 1.5 of GRUB and a file named core.img, but the image can't be seen because there is no file system there, it's just a 'gap')
  • Notes on "MBR Gap": Research indicates this "MBR Gap" can vary quite a bit depending on the OS. Even different versions of Windows have different size "Gaps". IE, the size is arbitrary and not set in stone. Lesson? Use a Partition Utility to look and see how things are arranged for an OS. It also seems that the GRUB 1.5 Stage is made to fit in 32K so it will fit into the MBR Gap. For OpenWRT, where the "MBR Gap" is about 256K, there's a LOT of unused space there. And no, didn't look to see exactly where in that space the CORE.IMG file.
• Boot Partition (contains grub.cfg file with settings and VMLINUZ (a compressed Kernel Image in "Boot Executable bzImage" format)
• Root Partition (all the files that comprise an OpenWRT installation, plus user added files, packages, settings, etc. on an EXT4 installation)
• Overlay "Partition" (only for SquashFS, not relevant to EXT4 based installations, offset exactly 3014656 Bytes from the beginning of the Root Partition, and addressed in more detail in a later section)

What to do / How to do it / Examples of Backing Up / Cloning an EXT4 Install of OpenWRT to an Image File

Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress);

• Syntax in one step;
  • dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img bs=512 count=511
• Syntax in two steps (optional);
  • MBR: dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img bs=512 count=1
  • "MBR Gap": dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img bs=512 skip=1 count=511

• Working Example (in one step): dd if=/dev/sdb of=/mnt/sda1/MBR_and_MBR-Gap.img conv=noerror bs=512 count=511 status=progress

Copy the Boot Partition (first partition on a 'stock' / 'standard' OpenWRT x86 installation):

• Syntax: dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img skip=511 conv=noerror bs=512 status=progress

• Working Example: dd if=/dev/sda of=/run/media/root/NTFS/Boot.img skip=511 conv=noerror bs=512 status=progress

Copy the Root Partition (second partition on a 'stock' / 'standard' OpenWRT x86 installation):

• Using DD;
  • Syntax: dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img skip=511 conv=noerror bs=512 status=progress
  • Working Example: dd if=/dev/sda of=/run/media/root/NTFS/Root.img skip=511 conv=noerror bs=512 status=progress
• Using CP;
  • cp -archive --force --verbose Source Destination
  • cp -a -f -v Source Destination
  • cp -afv Source Destination

...and finally, issue this command, just to make sure everything is flushed from cache / memory: sync

Restoring a Backed Up OpenWRT Installation from an Image File(s)

The reason it is done in separate stages is for demonstration purposes. And in a recovery situation, it might be that only one "Partition" needs to be recovered, instead of an entire Disk / SSD. Plus DD isn't very efficient when it comes to 'empty space', so doing a full backup image of a Disk / SSD can consume a lot of space (unless zeros are written to the empty spaces on the Partition and the output of DD is piped into some form of compressed file. And yes, there's that "sparse" option for DD, but it still has to scan / read the entire drive, including all the empty space (because DD essentially does a Sector by Sector backup instead of a file system based backup)

Assumptions

In syntax examples, if MBR and "MBR Gap" are cloned in one or two steps into one or two files (depending on user's choice), do it the same way on Restoring ("Assembly is the reverse of removal", in automotive repair manual vernacular).

What to do / How to do it / Examples of Restoring from an Image File

Copy the MBR and "MBR Gap" (Choice of doing it in one or two steps, it's also such short process there isn't really a need to add status=progress);

• Syntax in one step;
  • dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX bs=512
• Syntax in two steps (optional);
  • MBR: dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX bs=512
  • "MBR Gap": dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX bs=512 seek=1

• Working Example (in one step): dd if=/mnt/sda1/MBR_and_MBR-Gap.img of=/dev/sda conv=noerror bs=512 count=511 status=progress
  • Note: Theoretically the count=511 doesn't need to be included in the above example because the source file should be the same size, IE 0-511. But if one is working from a "whole disk / SSD" backup Image, that will stop DD's writing at Sector 511, thus not overwriting the first, second, and probably other Partitions.

Copy the Boot Partition (first partition on a 'stock' / 'standard' OpenWRT x86 installation):

• Syntax: dd if=/WhatEverPath/WhatEverFileName.img of=/dev/sdX seek=512 conv=noerror bs=512 status=progress

• Working Example: dd if=/run/media/root/NTFS/Boot.img of=/dev/sda seek=512 conv=noerror bs=512 status=progress

Copy the Root Partition (second partition on a 'stock' / 'standard' OpenWRT x86 installation):

• ...but first create the second Partition with CFDISK and format (mkfs.ext4 -L WhatEverLabel -v /dev/sdX ) it, if, that's if the second Partition doesn't already exist.
• Using DD;
  • Syntax: dd if=/dev/sdX of=/WhatEverPath/WhatEverFileName.img of=/dev/sdX seek=WhatEverOffset conv=noerror bs=512 status=progress
  • Working Example: dd if=/dev/sda of=/run/media/root/NTFS/Root.img skip=511 of=/dev/sda seek=34816 conv=noerror bs=512 status=progress
• Using CP;
  • cp -archive --force --verbose Source Destination
  • cp -a -f -v Source Destination
  • cp -afv Source Destination
  • Note: If moving a SquashFS based installation of OpenWRT, one can copy the files from the existing SquashFS version of OpenWRT over a standard / initial / virgin installation of OpenWRT, thus 'converting' from SquashFS to EXT4
  • Note 2: If doing the process in Note 1, keep in mind that there is also one thing to modify in the /boot/grub/grub.cfg file. Change this: rootfstype=squashfs To: rootfstype=ext4

...and finally, issue this command, just to make sure everything is flushed from cache / memory: sync

Syntax Notes

DD

• if: Input File (or Device)
• of: Output File (or Device)
• bs: "Chunk" size to copy from it= to of=
• conv=noerror: "Don't stop, just do what I told you to do, and don't give an excuse"
• progress = status: Show the progress of the copy process (only for a certain version of DD and up)
• conv=sync: Write all buffered blocks to disk (just to be safe, but can cause 'out of space' issues with MBR and "MBR Gap")
• skip: Skips X number of sectors for the input ( if= )
• seek: Skips X number of sectors for the output ( of= )

...and finally, DD seems to be a Zero Based utility, similar to how Disks (and other items like Intel SATA Ports, etc.) are numbered starting from Zero ( 0 ). So setting skip=511 will skip sectors 0 through 511, for a total of 512 sectors.

CP

• a: The purpose with this switch in the backup process is to preserve permissions, but the A actually stands for "archive"

...and finally, watch out for "Live" versions of Linux, like Rocky Linux, that have an "alias" setup for the cp command (which in the case of Rocky Linux is cp -i, where the -i=interactive). So what difference does that make? Well, when you type CP, you're actually typing CP -i. So if you type CP --force / -f, it won't work as anticipated and you'll end up having to answer yes to every 'overwrite a file situation' (or 'clobber' in the CP venacular). Why, because the -i switch always trumps the --force / f switch, so the command you're actually typing is CP -i --force / -f, and --force / -f loses to -i. And the final question of "Why would anyone do that with CP?" Well, it seems to be done in the name of "safety" (and hair pulling). And that's a great example of someone trying to think for you. It is now on this website's Top 3 List of the Most Bone Headed Stupid Things Ever.

Overlay, OverlayFS, Union Mount, SquashFS, F2FS, and more...

Real quick, here's what's going on with a typical Router based installation of OpenWRT that is stored on Flash Memory Storage.

OpenWRT is essentially installed on devices as if all of it's code were stored in an EPROM and just plugged into the board of a router. There are a bunch of abstraction layers that make this possible, both in the installation and function of OpenWRT, but that's what is happening.

How is it done? Using the Overlay / Union Mount capabilities of Linux. Sorry, not here to explain that fully, hence the above links.

Backing up the /overlay Directory

Ideally (since LVM isn't typically available to do "Volume Shadow Copies"), one would boot to an alternative OS, use DD to clone the /overlay "Partition" / "Device". Easy with an x86_64 based install of OpenWRT, but also very possible with a Router based on Flash Memory Storage too. For the Router scenario, again ideally, if the /overlay "Partition" / "Device" is on an external USB Flash Drive (as it should be if a router has a USB 3.0 port), just pop it into a "Linux Box" (virtual or otherwise) for backing up. It can also be done "live" without much chance of something being modified during the cloning process.

• Make a BackUp: While booted to an alternate OS (Rocky Linux, another OpenWRT install on a USB Flash Drive, etc.), in this example, booted to another OS, do... Wait, but first;
  • Locate where the /overlay Partition is;
    • LSBLK will show the connection between the /overlay Directory and what it is "connected to" (hint for x86 systems, look in the /boot/grub/grub.cfg file)
      • For an OpenWRT install with /overlay on a USB Flash Drive: /overlay will be on /dev/sdXy (where X is a letter and y is a number, example: sda1)
      • For an OpenWRT install with /overlay on a SATA Disk Drive / SSD: /overlay will be on a Loop Device, roughly equivalent to a VHD or VHDX File where a file represents a storage device
        • Use this command: losetup -a (it will display something like this: /dev/loop0: (Some Numbers) (/sda2), offset 3014656 (/dev/loop0 is the device and /sda2 is the "file", actually a real drive))
  • "Create" a LOOP Device (For an x86 (used interchangeably with x86_64, yes, yes, don't say it));
    • Generic Syntax: losetup -o OffsetInBytes /dev/loopX /WhatEverPath/WhatEverFileName
    • OpenWRT specific Syntax: losetup -o 3014656 -fP /WhatEverPath/WhatEverFileName (3014656* is the offset picked by the OpenWRT developers, losetup will pick the next available LOOP#)
    • OpenWRT Syntax Example: losetup -o 3014656 -fP /mnt/sda1/OpenWRT_BackUp.img
  • Mount the LOOP Device (again for an x86 system)
    • Example: mount -t f2fs /dev/loop1 /mnt/loop1 (mkdir /mnt/loop1 before this command of course)
  • ...and that's it for x86 devices. From there, just cp

*If you haven't spotted it yet, but you've been asking, "Where is that file that OpenWRT mounts a Directory named /overlay as a LOOP Device?" Answer: There is no file. It is simply a sector on the Disk / SSD Drive to the end of it. And you might ask, "But won't it be overwritten?" Answer: Nope, because everything before it is considered ROM (as in no write), so it won't expand. And yes that means the entire disk drive (well, at least the OpenWRT partitions) is considered a File that breaks down into a ROM Section and an /overlay Section.

Why 3014656 Bytes as the offset? Well, it works out to 5888 Sectors (512 Bytes / Sector, $1700 (that's Commodore Hexadecimal Notation)), 1472 Sectors (if translated to 2048 Bytes / Sector, $5CO), so nothing really jumps out. If it was the 1980s, there might be some unique point in the CHS format where this makes sense. Some other alignment? Doesn't make any sense with SSDs, but maybe something with HDDs. PI ( π )? Nope, that's 3.141592. Maximum "ROM" size (with room left for expansion)? That seems so much like Bill Gate's comment on 640K should probably be about enough for anyone, when putting important stuff above the top of RAM, thus blocking easy expansion and requiring workarounds.

mount -t f2fs /dev/loop1 /mnt/loop1

More on the LOSETUP Command

Create a "Device" to Mount, IE Associate a File with a Loop Device Node[edit | edit source]

losetup -fP WhatEverImageName.img (To see mounted "Devices": losetup -a), (f = File Name to configure as a Loop Device, P = Scan Partition Size, a = Show All Loop Devices, b = sector size)

"Man" Page for LOSETUP: https://man7.org/linux/man-pages/man8/losetup.8.html

Loop Device: https://en.wikipedia.org/wiki/Loop_device

Mount the Image[edit | edit source]

mount -t squashfs /dev/loop7 /mnt/squash

Keep in mind this only works with IMG Files created from Partitions. If an entire disk drive is imaged, the above method won't work. However...

Mount an Image of an Entire Disk[edit | edit source]

If an entire disk is imaged with DD, then a "map" of it can be viewed with CFDISK: cfdisk /dev/WhatEverLoopDevice (Example: cfdisk /dev/loop5) (Remember, use losetup -a to see "Devices")

Examining the output of CFDISK will show the names of the individual Partitions available: /dev/WhatEverLoopDevicePx (Example: /dev/loop6p1).

With the above information garnered from CFDISK, then mount individual Partitions: mount -t WhatEverFileSystem (squashfs, ext4, ntfs-3g, etc.) /dev/WhatEverLoopDevice /mnt/WhatEverMountPoint: mount -t squashfs /dev/loop6p2 /mnt/Loop6/p2 (Also: losetup -d /dev/WhatEverLoopDevice will delete the loop device)

Note on: LOSETUP -d WhatEverLoopDeviceName[edit | edit source]

Bad news. The delete command doesn't always work. After issuing the command on a legitimate Loop Device, and with no errors, the Loop Device will persist. Checking it with LOSETUP -a shows it is still there. Accessing it with CFDISK or MOUNT still works. Oh, well. Windows solution time. Reboot.

Example Uses for LOSETUP

• Syntax: losetup -o OFFSET_IN_BYTES /dev/loopX /path/to/your/file
• Example: losetup -o 3014656 -fP /mnt/sda1/OpenWRT_SDC2.img (this command allows for the LOSETUP command to select the next available LOOP Device in /dev)
• Mount the resulting LOOP Device created by LOSETUP: mount -t f2fs /dev/loop1 /mnt/loop1
• Displays LOOP Devices: losetup -a
• Delete LOOP Devices: losetup -d /dev/loopX

FAQs

• Why 511 sectors?
  • Use CFDISK (or any other Partitioning Utility) and you'll see that the first Sector of the first Partition starts at Sector 512. That is just the arrangement the OpenWRT developers chose. Research indicates this "MBR Gap" can vary quite a bit depending on the OS. Even different versions of Windows have different size "Gaps". IE, the size is arbitrary and not set in stone.
• Lessons about "MBR Gap"?
  • Use a Partition Utility to look and see how things are arranged. Research seems to indicate that the GRUB 1.5 Stage is made to fit in 32K.
• Why did the OpenWRT developers that focus so much on refinement and minimization use a whopping 256K for the MBR Gap?
  • The answer appears to be known only to them as there is no direct way to ask them. ChatGPT was asked and the answer was so bad, it was way worse than not knowing.

• Example of saving to a Disk / SSD from an Image File: dd if=/dev/sdb of=/mnt/sda1/MBR_and_MBR-Gap.img conv=noerror bs=512 count=511 status=progress

Copy the Second Partition (/root equivalent): dd if=/dev/sdX2 of=/dev/sdY2 bs=128M conv=noerror progress=status

Flush any buffered information to disk: sync

Other Notes

LVM (Logical Volume Management) -- Rough Draft

Signature (Disk Signature / Disk Identity): A 32 bit section of the MBR (GPT, DOS, LVM, etc). Oh, and CFDISK won't work as it does not facilitate LVM Disks.

• Steps for Creating a Bootable Disk with an LVM Partition
  • Create a BOOT Partition at an offset of 1M (equivalent to a quantity 2048 sectors that are 512 Bytes / Sector)
  • ...same steps as below, but for /dev/sdaX
• Steps for Creating an LVM Disk (NOTE: As in, that's an entire Disk, not just a Partition, and it assumes there's another disk to boot from, so do NOT do this, it is just here for reference)
  • PVCREATE (Create a Physical Disk, sort of equivalent to writing a Disk Signature): pvcreate /dev/sda (this will / can overwrite the DOS or GPT signature / identity of a disk)
  • VGCREATE (Create a Volume Group, nope, not sort of like creating a Partition (that's next), but rather like part B of PVCREATE, where one or more disks are assigned to a Volume Group): vgcreate VG1 /dev/sda (VG1 is an arbitrary name, it can be 'anything')
  • LVCREATE (Create a Logical Volume, equivalent to Partitioning);
    • lvcreate -L 16M -n BOOT VG1 (16M is the size of the first "Partition", BOOT is just a name given to the Logical Volume, it can be anything, VG1 is the name of the Volume Group from the previous step)
    • lvcreate -L 16G -n ROOT VG1
  • Mount (...but Format first) the Volume Group(s)
    • LVDISPLAY (shows the available LVs in /dev/NameOfVolumeGroup/NameOfLogicalVolume)
    • mkfs.ext4 -L BOOT /dev/VG1/BOOT -v (where -L BOOT is just the name of the Partition)
    • mnt /dev/VG1/BOOT /mnt/BOOT (mkdir /mnt/BOOT first)
• Steps for Creating an LVM "Partition" (IE, leave the MBR, "MBR Gap", and /boot Partitions intact.
  • ...same as above, but instead of referring to an entire drive (/dev/sda), refer to a partition (/dev/sda1)

GRUB

OpenWRT specifically mentions GRUB 2 in their documentation. But the way it works on an x86_64 EXT4 installation of OpenWRT would convince anyone that it is Legacy GRUB (except for the GNU GRUB version 2.02 heading at the top of the screen). The way it is configured via /etc/grub/grub.cfg would also convince anyone that Legacy GRUB was being used. So, the conclusion is that GRUB 2.02 is indeed being used, but since it is backwards compatible, it is in 'Legacy Mode'.

QEMU ( QEMU-NDB )

QEMU, specifically QEMU-NDB and QEMU-IMG, were commands available to install on OpenWRT that seemed like it would be useful in terms of VHD and other Virtual Disk Files, but were ultimately not needed.

Example Command: qemu-nbd -c /dev/nbd0 /usr/src/vmlinuz/vmlinuz

Rant on Backing Up OpenWRT (Not just the configuration files, but an entire 'Drive' (Flash, SSD, etc.))[edit | edit source]

How can the entire OpenWRT installation be backed up in a fashion similar to Acronis products, Clonezilla, etc.?

The first answer is that there is no built in facility to do that. Dang! Why not? Well, first of all, OpenWRT is intended to be installed on small router devices where this type of backup (Disk / Drive Imaging, Bare Metal BackUp, etc.) just isn't a thing. Second, and largely because of something mentioned in the first item, is that a program like that can't be crammed into a small enough space to fit on small router device.

Spoiler Alert: RESTIC[edit | edit source]

What about Baquacil, err, wait, that's a pool chemical. Bactine! No, that's the antiseptic. RESTIC! Yup, that's the utility for backing up in OpenWRT (the only one)

Sadly it's like EDLIN VS Microsoft Word. No offense to the nice RETIC people. It does it's job and works reliably. But seems as if was made in the 1980s (it was actually done in the 20 teens). Again, no offense, it has a ton of features and seems bullet proof.

And most of all, RESTIC is for backing up files on a Disk, Volume, or Partition. It is not capable of (nor is it intended for) backing up an entire disk or partition, including boot sectors, file tables, etc. It is not a Disk / SSD or Partition Imaging device.

Wah! But what if I want to do it anyway because... Well, because I want to

Other Choices[edit | edit source]

Clonezilla? Yup, great, but one issue. One has to boot from the Clonezilla CD / DVD / USB Flash Drive / ISO to do it. Router off, Clonezilla on.

Again; WAH! I want to do it with the Router still fully functional, IE from within OpenWRT.

What About DD?[edit | edit source]

Perfect! Except for one thing: Files might change right in the middle of a backup (because it takes a long time). Remember DD has no capability or concept of "Volume Shadow Copy" (Note: RESTIC doesn't have the capability, but it does have the concept of it, however that would require converting the entire OpenWRT install over to an LVM based Disk / SSD / Whatever 'Layout'.)

Ideal Objective[edit | edit source]

Ideally, the only solution would seem to be something like Acronis for Linux (or some other commercial product). Or something like RUBackUp and / or FSArchiver. The latter requiring the above noted conversion of OpenWRT over to an LVM based Disk / SSD / FlashDrive system. That's a lot of work. The best option would be making DD aware of Volume Snapshots and doing it that way. Ongoing work with this experiment will be to attempt changing the disk layout to include an LVM volume that allows for "Volume Shadow Copy" (Volume Snapshot, etc., IE, the Linux equivalent of that Windows feature) and using DD for a Bare Metal BackUp!

Cop Out, err, Respect![edit | edit source]

Please remember, none of this is rant is intended to be disrespectful to the OpenWRT, RESTIC, or anyone else that might have felt slighted. It is simply a colorful way of noting how in the software world, once someone is given something, the inevitable question and desire of 'What else can it do?" comes up. That's what this is about