I guess our experiences are different. It's not like I've cloned thousands of times, but I have never had an issue with a cloned drive whether the source was fully intact (as in going from a larger capacity drive to smaller, or vice versa) or showing the very earliest signs of failure (like the odd bad sector).
But, as I said, this is all about whether you can tolerate possible failure and having to start completely from scratch, or want to take the chance that this would be unnecessary. Based on my own experience, I choose the latter before resorting to the former. Others, yourself included, take the opposite approach.