Before the operating system starts killing processes when it is out of memory, consider signalling processes (somehow) to run their garbage collection now and release their freed memory back to the OS, assuming they are written in a garbage collected language. Need some standardized method of signalling.
Instead of every process having its own private garbage collector, provide it as an operating system service. This seems tricky, as each program may encode references differently.
Also consider signalling processes to do garbage collection when performance has gotten bad due to frequent swapping to virtual memory. This might not be too effective because stuff sitting around waiting to be garbage collected probably sits quietly swapped out. Doing garbage collection on them might even make things worse.