I will try to 'splain as best I can, as I understand it.
(Let me first say I am not an engine builder or even a long time hot-rod guy. Most of my experience is with old harleys, but I read and listen a lot. I never even built a car engine till last year, but I think I did OK: I'm very pleased with the results and my peers have told me they are impressed as well.)
The entire induction/exhaust system must be looked at as a whole. They must compliment each other to work efficiently. Efficiency = power.
Don't think of 'air flow' so much as 'pulses' and 'pressure waves.' At the mouth of the carb you have 14# of static air pressure. You open a valve to a vacuum (bear with me here for the sake of illustration) and the air column is not sucked in: it is pushed in. It is not a smooth flow. It is start-stop-start-stop because of the valves opening and closing. Pulsing.
Same thing on the exhaust side. The exhaust pulses in the pipes are a high pressure 'slug' followed by a low pressure 'blank'. This causes a high speed reverberation of pressure waves moving back and forth in the exhaust pipe system.
The idea is to get the pressure wave pulses of both intake and exhaust timed so as to enhance each other. You want a negative wave from the exhaust to be at the cylinder at the moment when the intake and exhaust valves are open (the overlap) to compliment the high pressure pulse in the intake manifold to help 'pull' more charge into the cylinder. This is 'scavenging.' This reverberation in a long exhaust tract does push some exhaust gasses back into the cylinder, but this is acceptable for the benefits it provides.
Your carb, manifold, cam, heads, & exhaust must all work in harmony to compliment each other.
One key element many people overlook is that the velocity of the gasses for both induction and exhaust is very important. Smaller intake runners and pipes typically work better on smaller less radical engines than big ones because a smaller mass moving faster can carry a stronger 'punch' than a larger mass moving slower. It also takes less energy and time to change it's movement: more efficient 'pulsing'.
However, when you are tuned for a full length exhaust system and you uncork it, the exhaust gas is straight out and you get less contamination of your intake charge due to the lack of back pressures during overlap. The contamination richens your mixture, when you remove it, you now are running lean. Long pipes give a wide torque band, but with that comes some contamination of the intake charge. (EGR: introduce exhaust gasses back into the cylinder for a richer mixture and you then lean out the carb for better emissions.)
There is an awful lot more than this to it, it is the subject of many volumes.
Luke, when you radically shortened your exhaust, you changed the scavenging effects the engine had been tuned for and thus the mixture being introduced into your cylinders.
That's my theory....Comments from those more knowledgable than I? I would like to hear any...