The open source alternative to Unity is UnrealEngine4. Epic's licensing is royalty based, if you make more than a certain amount per quarter. Unreal has both a mature AAA-grade toolkit, and a convenient graphical language for quickly prototyping before optimizing into the C++.
UnrealEngine4 isn't open source. Just because the source is available doesn't make something open source. The source code to Windows 2000 is available but it's definitely not open source.
The agreed definition is "Open-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose."[1]
There is a modern trend to redefine it as "something for which I can freely get the source code" but that ignores the original ideals of collaboration and cooperation.
