In this paper we investigate rogue waves over a chaotic background in the framework of higher order NLS equations that are relevant in deep water waves and nonlinear optical fiber applications. The following key features of rogue waves in deep water, as determined from our laboratory wave tank experiments and numerical experiments, are discussed: 1) The long time dynamics of the MI is chaotic and this chaotic background leads to enhanced rogue wave activity. Viewing the water wave dynamics as near-integrable, the chaotic wave train evolution is characterized by heteroclinic transitions in the Floquet spectrum of the NLS equation. 2) A nonlinear spectral decomposition of rogue waves in JONSWAP random sea states shows the proximity to instabilities and heteroclinic data is the main predictor of rogue wave occurrence.
The long distance dynamics of modulational instability in optical fibers is also studied. Periodically modulated cw waves are shown to evolve chaotically and generate rogue waves for a variety of physical parameters. Drawing parallels with our previous work on water waves we show that the chaotic evolution and rogue wave formation is characterized by heteroclinic transitions in the nonlinear spectrum. Different higher order terms are shown to be primarily responsible for the chaos and rogue waves in the parameter regimes considered.