This is pure random. There is a higher chance that while plundering something is damaged, but there is no guarantee that it will happen. If two buildings are broken you can consider having done something wrong and angered the game engine

Flo: are you sure that second one cant be destroy in second battle?
first battle - peshengs vs poland
second - poland vs peshengs

Im sure that kinzetka lost 2buildings (perhaps 3, but she is not sure)

So - there are a lot of possibilities where buildings of the fief can be destroyed:

1.
If you see the message: "armies loot fiefdom u Grande (no. 2315)" in the history than there was a battle and the shire was lost and this fief lost buildings.
Witold ist right... only fiefs with an owner.

2.
If there is no owner, the fiefdom will lose buildings (the fewer buildings on the fief the slower the process)

3.
If you see something like: "fire at fiefdom Ibelin (no. 3932)" in the history than a disaster happened to the shire and some fiefs lost buildings..... but only fiefs with owner.


On the whole it is better to take the fief back as soon as possible because you get silver out of it with what you can build destroyed buildings new. Especially if there are many buildings on the fief.