Digital images have been around us for decades now. As the quality and resolution of image have been improving, so has the increase of size of images. Several attempts have been made to improve the visual quality of lossy images without losing much of the data. Availability of large training dataset and increase in computing power has brought interest in the application of CNNs. Here, a powerful CNN is presented specific to the task of semantic image understanding to achieve higher visual quality in lossy compression. Though several JPEG encoding techniques may produce optimized image, these processes are ultimately unaware of the specific content of the image to be compressed. This method makes JPEG content-aware by designing and training a model to identify multiple semantic regions in a given image. A new CNN architecture is presented specifically to image compression, which generates a map containing semantically-salient regions so that they can be encoded at higher quality as compared to background regions