G&T programs are potentially disproportionate in terms of attention from the best teachers/classes, access to high-end resources, pipelines to the best colleges, etc. If those were available to everyone who could benefit from them, then there wouldn't be such competition for them.

Source: Am a parent of a G&T student and I see the intensity with which parents push their kids into the program.

I think the word "potentially" is carrying a lot of weight there.

Do you have any sources that these things happen and/or that they negatively impact students outside of those programs?

The argument I've heard previously is that G&T programs remove peer role models from classrooms, which lowers the class's average behavior and achievement.

I was a G&T student for 5 years (2nd through 6th) and have close relationships with people who were educators then administrators in a G&T program.