The bitterness could be from other things, such as hops of course… But also, 3 oz of vanilla is too much IMO and can add bitterness in high amounts in my experience.
I would not add more than 1 lb lactose in 5 gallons, total. You might already be there.
Perhaps just give the brew more time to mellow. Maybe needs 4-6 months or more.