You can't clear the cache for just one site, would be the short answer. At least as far as I'm aware.
The longer answer is, you shouldn't really care about clearing the browser cache for all sites. Your browser will eventually expire the cached files anyway, forcing you to download all the new files from every website eventually.
Keeping cookies and logons and passwords I would agree with, but cache isn't anything to be worried about keeping and modern browsers like Firefox and Chrome (even IE) allow you to clear cache without deleting your cookies and credentials anyway.
