OpenWRT is missing a big piece of the puzzle: configuration management and the ability to work with a "controller". OpenWRT is currently great at running stand-alone but has essentially zero support for being part of a "fleet" of devices managed centrally.
This means something as simple as changing the network name or password requires changing it on every single access point manually, and even worse if your mesh system relies on sharing frequently-changing state between devices.
OpenWISP tries to address this problem: https://openwisp.org - I suggest you check it out and solve the configuration management problem first.
The actual "mesh" part is actually relatively easy. Most commercial systems use basic Linux networking tools, HostAPd (sometimes with custom improvements, but this all ends up upstreamed or reimplemented upstream given enough time) and custom glue code to tie them together. A "mesh" system is typically a user-facing network being broadcast by all APs (with shared settings such as name and password) and an invisible, "backhaul" network each AP hosts (either on a separate interface or on the same interface as the AP - I believe some wireless cards can act both as AP and station as long as the channel is the same) and the other in the path connects to, and the glue code handles configuring all of that. 802.11s is also an option that can be used, and I'm pretty sure all of this is already possible to configure manually in Linux - what's lacking is the "glue code" to set up & manage all of this automatically.
This means something as simple as changing the network name or password requires changing it on every single access point manually, and even worse if your mesh system relies on sharing frequently-changing state between devices.
OpenWISP tries to address this problem: https://openwisp.org - I suggest you check it out and solve the configuration management problem first.
The actual "mesh" part is actually relatively easy. Most commercial systems use basic Linux networking tools, HostAPd (sometimes with custom improvements, but this all ends up upstreamed or reimplemented upstream given enough time) and custom glue code to tie them together. A "mesh" system is typically a user-facing network being broadcast by all APs (with shared settings such as name and password) and an invisible, "backhaul" network each AP hosts (either on a separate interface or on the same interface as the AP - I believe some wireless cards can act both as AP and station as long as the channel is the same) and the other in the path connects to, and the glue code handles configuring all of that. 802.11s is also an option that can be used, and I'm pretty sure all of this is already possible to configure manually in Linux - what's lacking is the "glue code" to set up & manage all of this automatically.