Broken shifter housing, can't seem to remove it
 

Hey guys, I've got a NA 5 speed with the separate shifter housing and it's broken in half. I've got the roll pin removed and the back half removed but the forward section of it with the four bolt holes will NOT come off of the transmission.

I know there are two alignment pins between the bolts, so obviously it needs to come up to come off of those before coming backwards off of the shaft/shift linkage/whatever the correct terminology is.. is there a trick to doing this? Am I missing something? Really would like to get this done with everything still in the car. I did manage to remove one from an extra transmission I had but that took a lot of prying and bullshit and I don't think I can do that with the transmission still in the car, also no idea how I would get the new piece back on in that situation.

I feel like I have to be missing something. Any help is appreciated. A picture of the part I'm talking about is attached.

Top be clear, the roll pin on the shaft inside of the housing has been removed, the piece that holds the end of the shifter is out, the whole back section of it is gone already. The only thing left is from the rear two bolt holes forward.