Characters with the ability to manipulate the environment won't be able to make changes within the castle since, as we have established, it is a living entity. The surrounding area cannot be changed, as well, since it's tied to the magic of the castle.