you've removed the buffer tube as well, right?
you should not have to remove the selector to get the box out. just put it in semi. wiggle the box out by gently pulling up on the front half and lifting at an angle... you can also apply pressure to the bottom part where the pistol grip was with your other thumb