A good read app note from ROHM on linear regulator On and Off characteristic. Link here (PDF)
When a linear regulator IC is turned ON, the electric charge is stored in the output capacitor and the output voltage increases up to the specified value. At this time, an inrush current flows from the input to output of the IC. The output voltage drops when the input power source is disconnected. This application note explains the series of operations when the power source is turned ON/OFF.