Alright, another couple of ideas:
1: What if when you reinforce the robber hideout, the robbers automatically attacks the troops you are reinforcing with with their full strength? This could either be instant or on a 1 hour delay allowing you to stack it up before they attack (though I think instant might be more fair.) You don't get the benefit of hiding behind a wall this way, but it could be a way to speed up the killing of robber hideouts where attackers won't benefit from it very easily. They would have to make def villages and use those to tank the hideout, but at that point they could also just build a second robber hideout hammer to clear it instead. Also, attackers would still have an advantage because they can siege it.
2: Make it so that robber hideouts scale more linearly with attacking strength and not have them be based mostly on resource production in the early game. The current robber hideout is very unforgiving for defensive players because they HAVE to build an economy to be able to build up their force since most of the time they can't rely on raiding because there is very few good defensive raiding units, and because of this, robber hideouts are a lot stronger for defensive players than what they can kill if they were to just attack it with their defensive army. Their losses will be several times larger than what a single village hammer offensive player will lose because their troops are spread out through several villages and because of this initial resource production scaling. Just either reduce the scaling or make it scale entirely off of the attacking strength of your troops to fix this issue.