Dorothy was the live in gf, the vicar's wife came before that, but yeah I agree, she surely noticed something was going on
But wasn't there a point where his mum and Dorothy threw everything in the back garden - smashing a lot of stuff in the process, being that she lived in a flat complex type thing, surely the neighbours would have heard or seen that? But I also think it's a case of not wanting to get involved with things that aren't your business, especially back then, I doubt it was very easy to contact social services and complain.