Re-caulking my basement shower but there a bigger gap between the shower and wall than I've encountered before.

So, this is a bit wider than I have ever run into before and it was filled with a combination of plaster and drywall "shims", which were pretty crumbly when I pulled them out.

It was clearly just filler. And it clearly needs some filler. I was going to use pink foam insulation board cut to shape to fill space before I recall and repaint.

Is that stupid? Is that a safe route to go?