Either paint over it or use that Magic eraser from that bald looking dude lol. It worked when I wrote with pernament marker on my walls.
first try a scraper for the fabric paint, if it peels the wall then just skim some joint comound on the peeled areas let it dry and sand with a 100 grit sandpaper and paint as usual
if any coloring remains test a small spot by paintig a spot and wait for it to dry , if the color comes through the paint after it is dried you will need to prime the wall first with a paint primerHow can I remove dirt from walls?
sand it off...How to clean flat painted walls?