You can exclude keys from secondaries. When you exclude a key from a secondary that includes a geometry, only the geometry is colour graded.
To exclude a key from a secondary:
If a key is included with the secondary, the keyer is enabled.
The key is excluded from the colour grade. The whole image or the areas defined by a geometry will be colour graded by the secondary.