It’s one of those things that some people care about and some don’t: if and when you should say “all of”, and when just “all” will do. Is there a rule about this? Well, it’s more of a well-established recommendation than a firm directive, and it’s generally understood that by following this rule of thumb you keep your prose sounding more lean and poetic.
In a nutshell, the time to use “all of” is before pronouns: “all of them”, “all of you”, “all of it” “all of me”, etc. You can’t really leave the of out in these cases; it just wouldn’t make sense without it. (“You can have all them.”) But “all” stands well on its own before everything else — i.e. any nouns and their articles or adjectives: “all the people”, “all the time”, “all his stuff”. There’s really no need for the of here.
There is an easy way to remember how it works Just think of these three famous songs with “all” in their titles: the jazz standard “All of me”, Pete Seeger’s “Where have all the flowers gone?”, and The Beatles’ “All my loving”. You can’t take the of out of “All of me”. But try dropping of into the second two song titles, and look what happens: “Where have all of the flowers gone?” “All of my loving”. Where has all the poetry gone?
If you can’t remember this guideline, or if you choose to ignore it or just don’t care, then you’re in good company. There’s one Pulitzer Prize-winning writer* who doesn’t follow this rule of thumb — but no-one’s complaining about his prose. Here are a couple of examples from one of his magnificent novels (guess the author and the book; the answer is below):
“He glanced around the car. Naturally, all of the other passengers were staring at them.”
“All of the grief and black wonder that he was never able to express, before or afterward, not to a navy psychiatrist, nor to a fellow drifter in some cheap hotel near Orlando, Florida, nor to his son, nor to any of those few who remained to love him when he finally returned to the world, all of it went into the queasy angles and stark compositions, the cross-hatchings and vast swaths of shadow, the distended and fractured and finely minced panels of his monstrous comic book.”
“All of me”, by Gerald Marks and Seymour Simons, sung by Billie Holiday
“Where have all the flowers gone?” by Pete Seeger, sung by Peter, Paul & Mary
“All my loving” by The Beatles
* Michael Chabon: The Amazing Adventures of Kavalier & Clay