Armor/builds question

Hello, I just recently started playing Witcher 3 (console if that matters). Just got to Skellige and I’m currently sitting at level 22.

My question is how much does the amount of damage stat for the swords and armor stat for armor pieces actually matter? I crafted and have been using the cat school armor set with the swords but as I progress further I’ve started to get equipment with higher damage and armor stats so I was wondering at what point it’s worth using the higher stat equipment but losing out on the bonuses for using the whole set.

In case it matters my play style is defiantly relying on fast attacks and signs so that’s been my priority in crafting my build.

Thanks in advance for the help!