I agree on losing all your robber offence would be a pain, and the infantry defence is more useful. Weird how little difference it makes.
Do you know what factors affect robber army size? I heard somewhere it was overall attack strength. If so then having the legionnaires would increase size of robber army... and resources/ treasures gained?