Obv I don't know what your crime was, but if you've no travel restrictions related to your CO then I don't think they can stop you - perhaps if they currently have you going in every week they could stifle it but maybe suggest phoning in ?
I'm no legal expert but I was on CO for 3 years and never had an issue travelling - went as far as Guatemala with no pushback. If I were you I'd get a script ready for your next session and ask him / her
- need a break
- time with your partner
- can come in just before anf just after etc etc
Hope that helps