OpenWRT Build System for CentOS 7

How does one create a "build system" (AKA Cross Compile / Cross-Compile / CrossCompile Build System) for OpenWRT?

Other Tutorials

• General Guide for building firmware: https://openwrt.org/docs/guide-user/additional-software/beginners-build-guide
  • Start of detailed Information: https://openwrt.org/docs/guide-developer/build-system/start
  • Pre-requisites for a : https://openwrt.org/docs/guide-developer/build-system/install-buildsystem
  • Some additional granular detail for working with specific versions: https://openwrt.org/docs/guide-developer/build-system/use-buildsystem
• Build environment tips: https://openwrt.org/docs/guide-developer/env
• SDK (for compiling specific packages): https://openwrt.org/docs/guide-developer/using_the_sdk
  • URL for SDK (look towards the bottom of the page for SDK): https://downloads.openwrt.org/releases/19.07.3/targets/mvebu/cortexa9/
• Third Party Site with nice information (Old, but this article itself notes even older OpenWRT documentation): http://www.acme-dot.com/building-openwrt-14-07-barrier-breaker-on-ubuntu-and-os-x/ (Don't follow the instructions as it GIT won't work anymore with non-SSL URLs)
• Entware Tips (The don't apply directly, but have some good tips if read from the context of OpenWRT)
  • Native Compilation: https://github.com/Entware/Entware-ng/wiki/Using-gcc-%28native-compilation%29
  • Compiling Packages from Source: https://github.com/Entware/Entware-ng/wiki/Compile-packages-from-sources
  • Creating Perl modules: https://github.com/Entware/Entware-ng/wiki/Self-installation-of-perl-modules

Environment

• Build System for Cross-Compiling: CentOS 7.8 (There seem to be some missing packages in CentOS 8, so CentOS 7.8 was chosen)
• OpenWRT 19.07.3 (Linux Kernel 4.14.180)
• Linksys WRT3200ACM Platform Information (see here for additional information)
  • Marvell Armada 385 88F6820 SoS (System on a Chip)
  • Manufacturer: Marvell, MVEBU (Marvell Embedded Business Unit)
  • Math Co-Processor: VFPv3-d16
  • CPU Information
    • ARM Family: Cortex-A (32-bit)
    • ARM Architecture: ARMv7-A
    • ARM Core: Cortex-A9
    • OpenWRT Target and Package Details
      • MVEBU (Marvell Embedded Business Unit)
        • Targets Available: cortexa53, cortexa72, cortexa9
        • Marvell Armada 37x/38x/XP (Armada 3700LP and Armada 7k/8k are 64 bit versions)
    • Equivalent DD-WRT / Entware Target and Package reference
      • ARMv7sf-k3.2: ARM version 7 with linux kernel 3.2
  • Sources:
    • https://en.wikipedia.org/wiki/List_of_ARM_microarchitectures
    • https://en.wikipedia.org/wiki/ARM_Cortex-A9
    • http://bin.entware.net/mipselsf-k3.4/

Steps (as of 9.2020 for the above Environment)

All of the below assumes one is storing files in the user's home directory.

OpenWRT recommends NOT compiling as a root users. To bypass this restriction;

• export FORCE_UNSAFE_CONFIGURE=1 (the export command adds the variable to the current user session, it will not persist if logging off, confirm with printenv or export -p)

Obtain Source Files from OpenWRT

• To clone into the current directory: git clone -b openwrt-19.07 https://github.com/openwrt/openwrt.git (downloads the 19.07 branch of OpenWRT, which also includes the latest version, 19.07.3)

• To clone into a specific directory: git clone -b openwrt-19.07 https://github.com/openwrt/openwrt.git WhatEverDirectoryName (downloads the 19.07 branch of OpenWRT, which also includes the latest version, 19.07.3)
  • To see repository, in web browser: https://git.openwrt.org/openwrt/
  • git tag (essentially shows available versions to build)
  • git branch (shows local versions of OpenWRT downloaded)
  • git branch -r (shows versions available for download)
• Feeds
  • Documentation: https://openwrt.org/docs/guide-developer/feeds
  • ~/openwrt/feeds.conf.default
  • ./scripts/feeds update -a (A script from OpenWRT to makes sure all files are up to date)
  • ./scripts/feeds (by itself, no "sub-commands", shows available options)
  • ./scripts/feeds list -n (shows main categories of feeds)
    • Should show these Names: packages, luci, routing, and telephony
  • ./scripts/feeds list -r OneOfTheFeedNames (shows the available packages, names are equivalent to what would be found with opkg list)
  • Example: ./scripts/feeds install htop (an enhanced version of top)
    • Files will be downloaded here: ~/openwrt/package/feeds/packages/, which are symbolic links to: ~/openwrt/feeds/packages/admin
• Checkout (not needed if a single branch like 19.07 is downloaded)
  • git checkout openwrt-19.07 (if multiple branches have been downloaded, switch to using this one)
  • Oddly, this command is documented using the latest version of OpenWRT as of 9.2020, but it doesn't work as expected: git checkout v19.07.03 (from https://openwrt.org/docs/guide-developer/build-system/use-buildsystem)
• SDK
  • Download the SDK and uncompress it (which is equivalent to the git clone command above)
    • URL for SDK (look towards the bottom of the page for SDK): https://downloads.openwrt.org/releases/19.07.3/targets/mvebu/cortexa9/
    • Don't worry about the name ending in x86_64 as this indicates the platform that the cross compiling will take place on
  • tar xf WhatEverFileName (be sure one is in the ~ directory)
  • OR... Do it manually (based on instructions from: https://openwrt.org/docs/guide-developer/helloworld/chapter1);
    • Remember, if the SDK has already been downloaded an error similar to: destination path X already exists and is not an empty directory.
    • ...so skip to the ./scripts/feeds update -a
      • git clone -b openwrt-19.07.3 https://github.com/openwrt/openwrt.git ~/openwrtSDK (NOTE: The instructions at the above address do not provide the correct URL when cloning)
      • git checkout v19.07.3 (If a specific version is specified there is no need to "checkout" a version as specified in the same instructions, plus it appears since the instructions have been written there are other naming conventions too, like openwrt-19.07, but choosing a specific version is better.)
      • make distclean wipes out all configuration, starts from a clean slate
      • ./scripts/feeds update -a: Downloads pa
  • ./scripts/feeds update -a
  • ./scripts/feeds install WhatEverPackageName
  • ToolChain Directory: ~/openwrt/staging_dir/
    • Linux ToolChains & Cross Compiling: https://elinux.org/Toolchains
    • ToolChain: https://en.wikipedia.org/wiki/Toolchain
    • Cross Compiler: https://en.wikipedia.org/wiki/Cross_compiler
    • musl: https://www.musl-libc.org/intro.html
    • EABI (Embedded Application Binary Interface): https://en.wikipedia.org/wiki/Application_binary_interface
• Package Building
  • Sources
    • https://openwrt.org/docs/guide-developer/package-policies
    • https://openwrt.org/docs/guide-developer/packages
    • https://openwrt.org/docs/guide-developer/helloworld/chapter3
  • Using the HTOP package as an example
    • Make a Manifest: https://openwrt.org/docs/guide-developer/helloworld/chapter3 (can copy similar to the below)
    • Find a "template" (similar existing package) and copy it: cp -R ~/openwrt/package/feeds/packages/htop ~/openwrt/package/feeds/packages/htopcustom (it's a symbolic link, so cp -R), modify and make changes
    • make menuconfig
      • PKG_NAME and PKG_VERSION need to change at minimum to get it to show up as a new option
    • disable all the Global settings to "build all"
    • Utilities (look in the Makefile file in the define Package/tarcustom (or WhatEverName) for where the package will be)
• Headers
  • Potential Locations;
    • ~/openwrt/build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/linux-mvebu_cortexa9/linux-4.14.195/user_headers/include
  • Sources
    • https://forum.openwrt.org/t/how-to-obtain-headers-for-openwrts-kernel/50737/6
    • https://openwrt.org/docs/guide-developer/packages

Environment Variables

export FORCE_UNSAFE_CONFIGURE=1

Standard Location: PATH=~/openwrt/staging_dir/host/bin:$PATH

For examples on OpenWRT "Hello World" Examples: export PATH=/home/buildbot/source/staging_dir/host/bin:$PATH

For persistent settings across reboots or logins: Add to ~/.bash_profile file

feeds.conf.default, feeds.conf, ~/openwrt/feeds.con, ~/openwrt/

Script that attempts to set Linux environment default variables for Make / Cmake: ~/openwrt/staging_dir/host/share/cmake-3.15/Modules/Platform/UnixPaths.cmake

Other package default path variables ~/openwrt/include/package-defaults.mk

• Created when git clone is used
  • Feeds Directory: ~/openwrt/feeds/packages/admin, utils, etc.
  • Feeds Symbolic Links in this Directory (for convenience?): ~/openwrt/package/feeds/packages/ (created when the ./scripts/feeds update -a command is used)

Notes on 9.7.2020

On Monday 9.7.2020 during the late afternoon (UTC-8) it was observed that OpenWRT 19.07.4 was released on Tuesday 9.8.2020. This would seem to indicate the server for OpenWRT downloads is somewhere in Europe.

• Downloads.OpenWRT.org: 176.9.48.73 is allocated to a company located in Germany (https://search.arin.net/rdap/?query=176.9.48.73)
• www.OpenWRT.org: 139.59.209.225 is allocated to a company in New York, United States (https://search.arin.net/rdap/?query=139.59.209.225)

The following interesting observations were made after the above noted event

• git tag : Lists all "major" and "minor" version branches of OpenWRT available.
  • v17.01.0
  • v17.01.0-rc1
  • v17.01.0-rc2
  • v17.01.1
  • v17.01.2
  • v17.01.3
  • v17.01.4
  • v17.01.5
  • v17.01.6
  • v17.01.7
  • v18.06.0
  • v18.06.0-rc1
  • v18.06.0-rc2
  • v18.06.1
  • v18.06.2
  • v18.06.3
  • v18.06.4
  • v18.06.5
  • v18.06.6
  • v18.06.7
  • v18.06.8
  • v19.07.0
  • v19.07.0-rc1
  • v19.07.0-rc2
  • v19.07.1
  • v19.07.2
  • v19.07.3
  • v19.07.4

• git branch -r : Shows available "major versions" of OpenWRT available.
  •   origin/HEAD -> origin/master
  •   origin/lede-17.01
  •   origin/master
  •   origin/openwrt-18.06
  •   origin/openwrt-19.07

• git clone -b v19.07.3 https://github.com/openwrt/openwrt.git produces the following output
  • Cloning into '/root/openwrtSDK'... remote: Enumerating objects: 13, done. remote: Counting objects: 100% (13/13), done. remote: Compressing objects: 100% (11/11), done. remote: Total 530753 (delta 0), reused 6 (delta 0), pack-reused 530740 Receiving objects: 100% (530753/530753), 183.81 MiB | 3.68 MiB/s, done. Resolving deltas: 100% (362276/362276), done. Note: checking out 'f3f38f40da74a193b4177be8d0fa746d00a753ea'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name
  • ...and git branch produces this output: fatal: Not a git repository (or any of the parent directories): .git. (Given that "HEAD" refers to a local copy of OpenWRT, this is not an issue)
• git clone -b openwrt-19.07 https://github.com/openwrt/openwrt.git produces the following output
  • Cloning into '/root/openwrt-19.07'... remote: Enumerating objects: 13, done. remote: Counting objects: 100% (13/13), done. remote: Compressing objects: 100% (11/11), done. remote: Total 530753 (delta 0), reused 6 (delta 0), pack-reused 530740 Receiving objects: 100% (530753/530753), 183.81 MiB | 3.69 MiB/s, done. Resolving deltas: 100% (362276/362276), done.
  • ...and git branch produces this output: * openwrt-19.07 (the asterisk indicates the "checked out" version if multiple versions existed)
• On 9.6.2020 this was contents of the feeds file included with cloning OpenWRT using openwrt-19.07;
  • src-git base https://git.openwrt.org/openwrt/openwrt.git;v19.07.3
  • src-git packages https://git.openwrt.org/feed/packages.git^2e6bd4cb86682b224803325127d3f777d40b3231
  • src-git luci https://git.openwrt.org/project/luci.git^fb2f36306756d0d0782dcab8413a8bb7ec379e54
  • src-git routing https://git.openwrt.org/feed/routing.git^3f8571194c2765ed31aa73459e86c2ebf943d27d
  • src-git telephony https://git.openwrt.org/feed/telephony.git^036cd451c35b82b3d8cac519864986894d9f6958

• On 9.7.2020 this was contents of the feeds file included with cloning OpenWRT with both the openwrt-19.07 and v19.07.3;
  • src-git packages https://git.openwrt.org/feed/packages.git;openwrt-19.07
  • src-git luci https://git.openwrt.org/project/luci.git;openwrt-19.07
  • src-git routing https://git.openwrt.org/feed/routing.git;openwrt-19.07
  • src-git telephony https://git.openwrt.org/feed/telephony.git;openwrt-19.07

Notes from 9.10.2020

When using the ./scripts/feeds update -a command, the First Top Tip is: The Directory defined in feeds.conf or feeds.conf.default with the src-link command / directive that contains the path to a feed must have a directory, subdirectory structure, Makefile.

If the Makefile file for a project named helloworld is located in /WhatEverPath/examples/helloworld/Makefile, the the following applies in the feeds.conf or feeds.conf.default file;

• Nope, won't work: src-link examples /WhatEverPath/examples/helloworld
• This will work: examples /WhatEverPath/examples
• Oh, and no trailing / (forward slash)
• If it isn't done correctly, this unhelpful error message will occur: ... *** target pattern contains no `%'.  Stop.
  • For the above error, some sites note an issue with leading or trailing spaces on any line (does not include spaces after a command / directive) and recommending TABs instead. But after testing with leading and trailing spaces in an attempt to produce the error, it appears that the OpenWRT developers corrected this issue within their script (IE, most errors due to spaces seem to have occurred in an era where only MAKE commands were used prior to the existence of the ./scripts/feeds script.
  • ...however, that above statement only applies to the OpenWRT scripts. The make command will cough up a hairball. And on another odd note, just getting rid of the spaces is not good enough as make expects tabs after some commands / directives. And to make it even better, when copying the example from OpenWRT, tabs will not be copied, spaces will be copied, and thankfully the OpenWRT documentation on this article (https://openwrt.org/docs/guide-developer/helloworld/chapter3) makes a nice rambling statement about spaces and tabs, when it should read something explicit like this: If you are going to copy and paste the below code, be sure to replace all leading spaces, short or otherwise with tabs. Why? Because make will not work with spaces AND it requires indentations in the form of TABS.
• Also in the feeds.conf or feeds.conf.default file do NOT use a ~ (tilde) in the path (like for a home directory) as it will not allow the ./scripts/feeds update -a command to work properly (plus it won't produce an error message, it just produces a zero byte index file.)
• And be sure to run the ./scripts/feeds update -a and ./scripts/feeds/ install -a command (or instead of -a for ALL, target a specific package

---

How do the variables in scripts from OpenWRT get their default values? (https://openwrt.org/docs/guide-developer/packages as noted here, they can be defined in the Makefile file)

pwd is a Linux command to get the current directory.

• ~openwrtSDK/feeds/WhatEverDirectory/MakeFile: Modifications here will be reflected in "make menuconfig"
• The symbolic links originally used in earlier examples are incomplete. There is NOT a directory to directory equivalent file / directory for symbolic links in ~/openwrtSDK/package/feeds. IE, the subfolders of all of the symbolic links in ~/openwrtSDK/package/feeds seem to be in a flat structure, where as the directories / files they point to are in a hierarchical structure.
• The README files in the various ~/openWRT/feeds directories make reference to the feeds.conf file which seems to now be titled feeds.conf.default. Documentation seems to indicate either one of these file names will work, although the one ending in .default seems to trump feeds.conf if both are present.
• In order for a new item to show up in make menuconfig: Add it to /
  • NO ./scripts/feeds/update needs to be run, if: something is manually added as a symbolic link.
  • OR
  • run the ./scripts/feeds install -a command (obviously this is a better choice)
• Error Logs are located: ~/openwrtSDK/logs/package/feeds/
• If duplicating a directory, start by changing the define Package/WhatEverName that contains: SECTION, TITLE, etc. and every other reference to the same WhatEverName
  • It was noticed at minimum, the last line: $(eval $(call BuildPackage,WhatEverName)) had to be changed too.

Notes from 9.11.2020

• Name of Package in Makefile
  • define Package/openssl-custom
  • define Package/openssl-custom/description
  • define Package/openssl-custom/install
  • $(eval $(call BuildPackage,openssl-custom))
• Name of Directory in "feedscustom" must match items in Makefile
  • source "$(SOURCE)/Config.in"

Notes on 09.13.2020

SOLUTION for the below: (from: https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem ):

• ...CD to the SCRIPTS Directory from the "GIT Cloned" version of OpenWRT
  • ./feeds update -a
  • ./feeds install -a

...all fixed. Naturally installing 'feeds' solves the issue of missing dependencies, etc.

Here's a stupid error that makes complete sense, yet shouldn't even exist in this situation: Makefile 'package/linux/Makefile' has a dependency on 'kmod-phy-bcm-ns-usb2', which does not exist

What's the situation? This is a build for a WRT3200ACM, with no Broadcom chips. So what the fuck package is in the official SDK for this same router that has a dependency like this?

Oh, and better yet, what about the below?

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'r8169-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'e100-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'bnx2-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'ar3k-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'mwifiex-sdio-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'kmod-phy-bcm-ns-usb2', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'edgeport-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'kmod-phy-bcm-ns-usb3', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'amdgpu-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'radeon-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'prism54-firmware', which does not exist

WARNING: Makefile 'package/linux/Makefile' has a dependency on 'rtl8192su-firmware', which does not exist

Of course none of this shit exists, it's a GOD DAMNED SDK for the WRT3200ACM!

There is no solution to all the above errors. Just tolerate it amongst all the other fucked up shit.

Also, do NOT delete any of the newly created Directories in the ......feeds Directory as they are actually symbolic links to WhatEver custom Feeds Directory structure was created.

And if one is looking for the OpenSSL package, it is of course intuitively named openssl-util. And following that very clear path, the Makefile file for openssl is located in /feeds/base/package/libs/openssl

09.13.2013 - Less Vitriolic Notes

git clone -b openwrt-19.07 https://github.com/openwrt/openwrt.git /root/openwrtSDK.Manual: Copies the 19.07 (IE, the lateversion of files from the OpenWRT sources at GitHub

git checkout v19.07.3

make distclean: https://stackoverflow.com/questions/51837675/what-are-the-differences-between-make-clean-make-clobber-make-distclean-make

./scripts/feeds update -a: Downloads all the v19.07.3 Makefiles and associated "stuff"

./scripts/feeds install -a: Creates indexes used by "make menuconfig", which in turn creates the visual selections within menuconfig script.

make menuconfig: run the menuconfig script (the Build the OpenWrt SDK will create an SDK PACKAGE)

make toolchain/install V=sc: use the toolchain/Config.in file and install it (see the Config.in file in the root openwrt... directory, then the /toolchain/Config.in file), the -V=sc enables verbose output with details of what it is compiling instead of a boring command prompt.

staging_dir/host: Seems to contain x86_64 files that are "translated" to other platforms

Another observation that seems a bit obvious, but isn't really explicitly stated anywhere is that the "make menuconfig" "script" settings apply only in certain circumstances. For instance, when using the above make... ...toolchain/config command, there were no packages compiled and put into a bin directory. As a test, selected the openssl package (which includes libssl) to be created as a modular package. And it wasn't created, nor should it have been.