The upgrade for shelter and wall is a defensive bonus in a battle, always for the one who is attacked!
The attacker can lower this bonus by having a high skill value. His dexterity makes it easier for him to penetrate the enemy shelter.
Each battle report now also shows the skill value of the attacker and the defence bonus for the defender, which is credited to the attacker's shelter based on the skill value of the attacker!
When the bonus does not apply
Of course, you don't get the defence bonus if you are at the cemetery or hunting, but really only if you are actually in the shelter.