I feel like it's similar to image compression, you lose a bit every iteration. Consider that the original model was weighted towards common aspects across the training set. Even with some creative prompting for your source images you could unintentionally introduce bias and reduce variations across images generated by your new model. You also get any mistakes or inconsistencies baked in.