Color swatch window doesn't change color of selected object

I noticed that when selecting an object, choosing Color from the contextual menu the color window (hurray for using the standard color window of Mac OS X!) pops up and stays there even after selecting another object.
But when you then change something in the color window, instead of changing the color of the newly selected object, it changes the color of the object that was selected when the color window appeared.
To me, this is confusing because there is no relation anymore between the first object and the color window, so after selecting a bunch of objects it's unclear which object the color window belongs to.

So after changing the color of the first object and selecting the second object, I would have to close the color window, right-click the second object and choose color, even though the object was already selected and the color window was already there!

(Note: I have observed this behavior in other software too, so maybe it's something OS X related that can't be changed).