Yes it is when we do not have a proper system in managing them. Most importantly is to have the right number of personnels (resources) and competent ones to get the performance report coming your way for all those projects under your portfolio. Managing at portfolio levels do need more attention to the kind of risks, issues and stakeholders need, thus your communication plan must be no less than perfect.
Other than that managing your own character is as important because you are dealing at high level. Your compassion and passion in day to day managing must be strong and consistent. (must be a human person if u know what i mean) That is the reason leadership is so important to a portfolio manager.