The objective of this thesis is to advance the mathematical theory of Bayesian Neural Networks for both Shallow and Deep Networks. These kind of networks are very promising since their flexibility in modeling the phenomenon under study as well as to provide accuracy measures of the estimated quantities a problem not well understood in classic network theory. Computational aspects would another aspect cover by this thesis since standard methods for network training will not be suitable for real applications. The other objective of this thesis is the applications of the new methodology to important problems in Health Sciences for e.g. Omics or  Bioimaging data. The study will involve expertise in several topics including Bayesian non parametrics methods, information geometry, Monte Carlo methods as well as standard methods in Deep Learning. The developed methods will be tested on different life sciences datasets (separatly and integrated) such as EHR, Omics and bioimaging (e.g. CT, PET, digital pathology), both publicly available and originally produced by DSH partner labs, on different tasks.