It's not like they keep annual census in that vast field of snow and ice, dude.
What IS known is 100,000 of them (including men, women, elderly, and children) decided to unite under Mance Rayder. Only about 20,000 of them are capable fighters, who participated in the Battle of Castle Black.
Not everyone of those 100K had time to migrate South with Lord Commander Jon Snow's blessings, and they are currently being absorbed into the Dead Army, along with the thousands who declined to join Mance's camp in the first place.
And maybe everyone who was ever buried too. That's an even bigger number.