what is Rayleigh Taylor instability and how ultimately it results from photon gas generation?

Answer 1

The R-T instability is the result of a heavy fluid supported by a light fluid in a gravitational field or, equivalently, during acceleration of the fluid interface. The classic demonstration is to fill a glass with water, cover it with a flat plate, turn it upside down, and slide away the plate. Theoretically, air pressure will support the water (keep it from falling) if the interface is perfectly flat. However, it's an unstable equilibrium, which means the slightest perturbation in the surface will grow exponentially until the water falls out.

R-T is a ubiquitous phenomenon in driven fluid systems such as Inertial Confinement Fusion. The most popular ICF approach is to laser heat the interior of a holraum to create an intense electromagnetic radiation field. Perhaps this is what you mean by "photon gas". This can act like a light gas accelerating the ICF target inward, with a resultant instability. Photon pressure itself, however, only has a negligible effect in this case. It's the shock wave from the explosive radiation ablation of the target surface that supplies the drive pressure. Though closely related to the R-T instability, this is referred to as a Richtmyer Meshkoff instability.