The first experimental observation of spin-orbit (SO) coupling in Bose Einstein Con densates (BECs) provided an interesting new platform to explore a fascinating and growing field of research and lead to rich physical e↵ects. In ultracold atomic sys tems,the synthetic SO coupling can be generated using two counter-propagating Ra man lasers that couple two hyperfine ground states. Motivated by these experimental findings, some theoretical activities have been committed to the physics of SO-coupled BECs under di↵erent conditions. In this thesis, we explore the nonlinear dynamics induced by the modulational instability (MI) in dissipative and non-dissipative SO coupled BECs in free space. The first chapter gives the general introduction of BEC and reviews of the basics and essential concepts used throughout the thesis: the Gross Pitaevskii (GP) equation, SO coupling, solitons and MI process. In the second chapter, our investigations start with the derivation of a new vector form of the cubic complex Ginzburg-Landau (CGL) equation describing the dynam ics of dissipative solitons in the two-component helicoidal SO coupled open BECs. Employing standard linear stability analysis, we analyze theoretically the stability of continuous-wave solutions and obtain an expression for MI gain spectrum. Using di rect simulations of the Fourier space, we numerically investigate the dynamics of the MI in the presence of helicoidal SO coupling. The validity of the analytical solutions obtained is confirmed by the numerical simulations. In the third chapter, we report the dynamics of the MI process, exclusively studied in a two-component BEC with Rashba-Dresselhaus (RD) SO and helicoidal SO cou plings. A generalized set of two-dimensional GP equations are derived. The tunability of the helicoidal gauge potential is exploited to separately address BECs dynamics in free space and a square lattice. The MI growth rate is derived for each case, and parametric analyses of MI show dependence of the instability to interatomic interac tion strengths, the RD SO coupling, and helicoidal SO coupling, which combines the gauge amplitude and the helicoidal gauge potential. Direct numerical simulations are performed to confirm the analytical predictions. Trains of solitons are obtained, and their behaviors are debated when the RD SO parameters are varied under di↵erent combinations between the gauge amplitude and the helicoidal gauge potential. The latter gives a potential way to manipulate the trapping capacities of the proposed BEC models. In conclusion, the results and discussions are presented. The scope for future work has also been suggested in detail.