Let be the velocities of the balloon and man w.r.t. ground.
Therefore velocity of man w.r.t. balloon is,
Initially, the whole system is at rest.
Therefore the center of mass C of system is at rest.
When the man starts climbing up the ladder, he exerts force on the ladder in downward direction and he gets an equal and opposite reaction in upward direction.
Since, there is no external force on the system, therefore when the man starts climbing up, the center of mass of system remains at rest at C.
From equation (1) and (2), we have
Negative velocity indicates that the ladder will move down.