Recognizing Traffic Signs using intelligent systems can drastically reduce the number of accidents happening worldwide. With the arrival of Self-driving cars it has become a staple challenge to solve the automatic recognition of Traffic and Hand-held signs in the major streets. Various machine learning techniques like Random Forest, SVM as well as deep learning models have been proposed for classifying traffic signs. Though they reach state-of-the-art performance on a particular dataset, yet fall short of tackling multiple Traffic Sign Recognition benchmarks. In this paper, we propose a novel and one-for-all architecture that aces multiple benchmarks with a better overall score than the state-of-the-art architectures. Our model is made of residual convolutional blocks with hierarchical dilated skip connections joined in steps. Intrinsically, our model achieves 99.33% Accuracy in German traffic sign recognition benchmark and 99.17% Accuracy in Belgian traffic sign classification benchmark, while classifying traffic signs in real time. Moreover, we propose a newly devised dilated residual learning representation technique which is very low in both memory and computational complexity.