I've been hit in the face many times, and never had one come out. I've opened my eyes under water, and they didn't come out. The only time I lost one was surfing - it was really windy, like howling, and when I punched out the back of a wave paddling out, opened my eyes, and the wind peeled it off my eye. It was weird.
Mine are hard to get out too. Maybe you guys don't have them fit properly.