Supervised learning is a popular mechanism to teach machines vision-based tasks and skills. However, human supervision is a bottleneck for building generic machines that can operate across different contexts, environments and applications. Ideally, machines should develop their own effective and possibly creative strategies for using the sensed data and their experience to continually learn without humans at their side. The research activities related to this PhD position will focus on building novel deep learning-based vision algorithms to teach machines to seamlessly understand environments through 2D or 3D perception.