I believe the castle upgrade is cosmetic only. Too damn expensive, so i haven't given it much thought.
Here is what the Wiki says: Modifies the front and stairs inside the castle,
giving you attack points to the side of the gate and a balcony above it
And the wiki isn't *quite* correct, or at least leaves out some information.
The stone outer wall upgrade gives you four additional foundation (large) points outside the walls, so you can move some buildings that aren't essential to siege outside the walls--buff towers and such--and replace them inside the walls with more useful buildings, such as processing factories, farm, drill grounds, et cetera, where they are better protected.
Remember, each proc provides five (5) crystal extractors, and there's no question that more extractors are definitely better--more crystals available for buying defensive towers, offensive and defensive skill upgrades, scrolls, and so on.