Most folks use super glue to put inserts in because it is fast and easy. They usually get the results you have experienced.
Wrap 40 or 60 grit sandpaper around a small screwdriver. Put it in the end of your arrow and twist around to roughen up the surface. Then clean with alcohol.
do the same with the inserts. Then get some good epoxy, or bohning insert glue (24 hour kind). Glue them up.
Should fix your problem.