Is town centre really as bad as people say?

I’ve (f) just moved back to look after my dad part time. My sister has told me to never walk through town at night alone. I must admit when I walked from the station to Tower Ramparts on a Monday evening it wasn’t particularly busy, lots of men hanging out by their cars and hollering at each other- but I didn’t think it was that bad. I work in London and have lived in various areas that are considered ‘rough’ there. I haven’t often felt uncomfortable in London but all my family are ‘warning’ me about Ipswich.