In a society with polygamy wealthier, older men tend to have more wives, usually young wives, which leaves many younger men without a partner and tends to result in these young, single men acting out more. It essentially shuts out a section of the adult male population from marriage. Polygamy made more sense in antiquity when men died more often and earlier, which would leave women without partners if not for plural marriages. This is why plural marriages of the past were most often men with multiple wives and not the reverse. This effect can actually be exacerbated by what is called the "lost boys" phenomena in which older men in the community will persecute and ostracize younger men because they're competition for the younger women which in turn could lead those younger men to turn to unsavory activities or just having poorer outcomes in general.
Activist judges.
But seriously, we, as a society, have judged gay marriage to be legitimate. The thing is a lot of the arguments in support of gay marriage and against maintaining a rigid traditional definition(marriage is not about reproduction, consenting adults should be able to for relationships with each other etc) do in fact work for other forms of marriage that have yet to be recognized.
I'm not saying this to knock gay marriage. If anything, for the reasons I listed above in response to the other poster, polygamy is much more likely to negatively effect me, as a young man with little money, than gay marriage is but I'm still interested in at least having the discussion about it.
For better or for worse, the gay marriage advocates really gave any potential incest advocates a hand when they so effectively argued that marriage is divorced from reproduction
