The defense increase is fairly minuscule and won't turn any actual fights around, though it can be effective against small raids. If you like you can examine the defense points in the battle simulator, but for reference the lvl 10 provides 200 defense points which is about the same as 4-5 unupgraded phalanx and the lvl 20 version is 800 points (worth about 16 to 20 phalanx). This power is of course boosted by walls and ditch but as you might imagine having the effective power of maybe 10 fully upgraded and hero boosted phalanx isn't much.
That being said residence has other benefits
1) it produces nice culture
2) it makes your village more resistance to being chiefed (the enemy has to destroy the residence first)
3) if you later need the residence to make settlers or chiefs or such you have to rebuild it and you can't gold it up either
Residence is usually quite high on my "replace when needed" list though as the chief protection is questionable at best and the culture production is nice but usually by that point I don't mind losing some culture generation to put up a granary or a great stable for example. The third benefit is only useful if you have free slots or plan to chief from yourself at some point.
My verdict: Remove if you need the space
Other buildings you might want to remove are:
Smithy (once you have researched all the upgrades you need)
Academy (once all the units are built)
Crannies (if any)