If you just delete the house sign (assuming it's owned), it'll go back to an unowned state.
And then you delete the townhouse sign.
That's all there *should* be to it, but I don't know what version you're running, mods you've made to it, etc.
Crash logs help. :)