I think you're right to an extent. There's no doubt that the Romans (post Marius reforms) where one of the toughest sobs ever. These were professional soldiers, fighting was their living. I mean we're talking about guys that (while on campaign) marched for 20kms with 30kgs on their back! And when reaching their destination they didn't catch a break, they had to build a freaking mini fortress with palisades and all. Every single night.
BUT, I think people are talking about single battle, one guy against another guy. The roman legionary, his equipments, his tactics, everything about him was for the purpose of fighting in groups. When their formation was broken, they didn't do well (I know, you can say the same thing about any army). Look what happened to them in the battle of the Teutoburg Forest. That day they didn't have the benefit of their maniple formation, they had instead to fight a bunch of small, single battles and got slaughtered by Arminius and his Germanic warriors.
The Romans were not THAT amazing in single combat, not because they were not tough sobs, 'cause they were, but because that is not what they were trained to do.