How is the second half of the show?

I watched S1 when it first came out and recently rewatched. In my opinion it's a 10/10 Season for a show and try as I might, I can't think of a single thing I'd change or that doesn't add up.

I went on to watch S2 and it's a noticeable drop-off for me. I wouldn't go so far as to call it bad, but especially in direct comparison to the first season it seems very messy in terms of writing. I could go into some details, but that's not really the point of my post.

My question is: how does S3 and S4 stack up in comparison to the first two? Because I'm a bit on the fence about watching on.

TL;DR: big fan of S1, found S2 pretty meh (storywise). Is it worth to watch on or just keep the fond memories I have?