Using an artifact with the Lust trait: “This creature has access to all the enemies’ non-temporary Spell gems”. It doesn’t grant you any gems.
I’ve also noticed gaining traits temporarily (e.g. with “soul to keep” trait) that should grant access to a certain selection of gems (e.g. the grimoires) doesn’t give you access to them.
0.0.35 Win 10