Instability due to weight?

I’ve recently stepped up to a decent Ridley Fenix SL frame and added some carbon wheels.

I was highway excited to ride it until my first descent where it developed a large front end shake which nearly threw me off the bike.

I’ve ridden the descent many times and never had this issue.

Tested the bike with different wheels and had no issues what so ever yet trying 2 pairs of carbon wheels both cause shake/wobble when I descend.

Is it possible that a frame/wheel combo can be too light for a riders style and adding extra front end weight solves the issue?

Thanks