Although Clint has star billing, Wallach's performance is the glue that holds it all together.
Yes, you're 100% right about that. Wallach is the glue: the gravy in the dinner. It's only when he's on screen that you know the main story is in progress. Not one bad casting decision though and it wouldn't be as good without any one of them.
...even if the casting decisions were partly "those who agreed to be in it".