Simon Cowell may be wrong about a lot of things -- like, you know, his penchant for V-neck shirts -- but one thing he got a billion percent right was assembling One Direction on 'X Factor U.K.' Since then, the cantankerous Brit has kept in touch with the boys and reveals how he keeps them in line!

Cowell admitted to Hollywood Life that the massive, meteoric rise to fame that 1D experienced since their 'X Factor' days can be disastrous for some stars, because it's a whole lot to handle -- especially considering the boys' work load. They're on their third album in two years!

"A third album can be tricky because they've had so much success and are so young and they can turn into nightmares," Cowell said. "I see that in the media all the time when kids get out of control, but they are good."

Phew! He also said he has no plans to try to control the band in terms of creative direction, which will likely prevent them from going off the deep end -- they'll have some freedom to grow! "I have a good relationship with them," Cowell explained. "From the beginning, I wasn't going to be some awful Svengali telling them what to do. We are all smart and they can work it out themselves and they have worked it out themselves and they have fun. I've always said to them, 'If it’s not fun, then stop doing it.'"

Based on the giggles they inspire with their videos and antics and how sweet they are to their beloved Directioners, we bet they're having plenty of fun -- so fans can rest easy!