There are a few things that could cause this but my first guess would be that your arrows are too stiff of spine. Put some heavier field points on to lower the arrow spine…or try lighter spined arrows. The biggest challenge with selfbows for me is arrow spine, they sometimes have a very narrow spine requirement, just a couple of pounds can make a difference. By comparison my center shot recurves will shoot a 20# range of spine.
You could be torqueing the bow, brace height might be too low, but again I’d bet arrow spine is the culprit. Have fun.