This paper is concerned with the controllability of higher order fractional damped delay dynamical systems with time-varying multiple delays in control, which involve Caputo derivatives of any different orders. A necessary and sufficient condition for the controllability of linear fractional damped delay dynamical systems is obtained by using the Gramian matrix. Sufficient conditions for controllability of the corresponding nonlinear damped delay dynamical system are established by the successive approximation technique. Examples have been provided to verify the results.