I was reading this over again and being a person that also likes to tinker (some, or most, of the time mine turn out to be a big waste of time and cranial calories) and wondered what if you either sand or turn on a lathe or similar, the back 1/2″ or so of the 4″ sleeve to a fine thin edge. This would ease the bump on the draw.
Another thought would be to lace up the shaft behind the sleeve to build it up to sleeve dia. I’d suggest using a fine nylon sewing thread, since this would make the transition easier. Then seal it with fletching glue or similar.
What would add a HUGE amount of strength, in my mind anyway, is finding a way to put a taper on the front of the sleeve to glue the head directly to the sleeve, then slide the sleeve over the shaft.
Anyway, those are my thoughts, hopefully you figure out something.