In the applications where you typically find voltage regulators, you find two capacitors in parallel with the load.
Their purpose is to smooth out voltage fluctuations and pulses. Therefore they are referred to as smoothing capacitors.
The capacity of the capacitors usually amount to several μF. (In order to size the capacitors correctly, you have to know the fluctuations, pulses as well as the load very well.)
The next step you protect the voltage regulator ...