In general, semantics concerns the extraction of meaning from data. Semantic vision seeks to understand not only what objects are present in an image but, perhaps even more importantly, the relationship between those objects.
In semantic vision, an image is typically segmented into regions of interest. Each segment is allocated a classification so that all pixels in an image are assigned a class (e.g. car, road, footpath, person, tree, sky).
Contextual relationships provide important cues for understanding a scene. Spatial context can be formulated in terms of the relationship between an object and its neighbouring objects. For example, a car is likely to appear above a road, but is unlikely to be surrounded by sky. Contextual relationships can also be inferred from the relationship between items in an image and the camera. For instance, if a tree is occluded by a car, then the car is closer to the camera than the tree. The ability to attribute relationships between objects demonstrates reasoning, an important step towards true “cognition”.
Imagery is routinely collected by satellite, aerial surveillance (from both aircraft and drones) and ground-level cameras. By reviewing images of the same area that are collected at over time, it is possible to identify and track changes. Identifying significant changes in a set of images is not a trivial problem. Seasonal variations and local weather conditions (such as fog, rain and cloud shadowing) can be falsely interpreted as change. Semantic vision can transform visual images into descriptions of the world; providing a more robust foundation for change tracking.
Applications for semantic vision include autonomous driving, medical imaging analysis, industrial inspection, classification of terrain from satellite imagery and data management.
Using the AIML-developed RefineNet to undertake semantic segmentation of scenes from the CityScapes dataset (video prepared by Anton Milan).
Competitions and achievements
AIML has won numerous global competitions in semantic vision and has made major contributions to the development of the methodology.
2nd place Robust Reading Challenge on Reading Chinese Text on Signboard (ICDAR 2019) Learn more 3rd place AI edge contest (image segmentation) Learn More 1st Place
Nuclei images segmentation challenge (MICCAI 2018)
Learn more 1st place Retinal fundus glaucoma segmentation challenge (MICCAI 2018) Learn more 4th place JD Fashion Item Search Competition (2108) Learn more 1st place Cityscapes semantic image segmentation challenge (2016) Learn more 1st place ImageNet challenge for the task of Scene Parsing (2016) Learn more 1st place Pascal VOC semantic image segmentation challenge (2016) Learn more 1st place “Focused Scene Text” for the task of End-to-End Scene Text Recognition (ICDAR 2015, Challenge 2) Learn more
2019 “Knowledge adaption for efficient semantic segmentation” Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, Youliang Yan
We present a novel knowledge distil framework tailored for semantic segmentation. Our method achieves superior results with significantly reduced computation overheads.
2019 “One shot segmentation: unifying rigid detection and non-rigid segmentation using elastic regularization” Jacinto C. Nascimento, Gustavo Carneiro
We propose a novel deep learning method for object segmentation with low training complexity that needs small training sets. These two advantages speed up the training time when compared to recent state-of-the-art approaches. We show that it is possible to have a machine learning-based segmentation system that operates directly on the space combining rigid and non-rigid deformations.
2017 “RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation” Guosheng Lin, Anton Milan, Chunhua Shen, Ian Reid
We present RefineNet, a novel multi-path refinement network for semantic segmentation and object parsing. The cascaded architecture is able to effectively combine high-level semantics and low-level features to produce high-resolution segmentation maps. Our design choices are inspired by the idea of identity mapping, which facilitates gradient propagation across long-range connections and thus enables effective end-to-end learning. Experiments show that our method sets a new mark for state of the art in semantic labelling.
2016 “Efficient piecewise training of deep structured models for semantic segmentation” Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel
We propose a method which combines Convolutional Neural Networks (CNNs) and Conditional Random Fields (CRFs) to exploit complex contextual information for semantic image segmentation. We formulate CNN based pairwise potentials for modelling semantic relations between image regions. Our method shows best performance on several popular datasets including the PASCAL VOC 2012 dataset.
2016 “Wider or Deeper: Revisiting the ResNet Model for Visual Recognition” Zifeng Wu, Chunhua Shen, and Anton van den Hengel
The trend towards increasingly deep neural networks has been driven by a general observation that increasing depth increases the performance of a network. We analyse the operation of deep residual network architectures and use our results to derive a new, shallower, architecture of residual networks which significantly outperforms much deeper models. Our architecture outperforms the previous very deep residual networks not only on the ImageNet classification dataset, but also when applied to semantic image segmentation.
Deep Reinforcement Learning for the Active Extraction and Visualisation of Optimal Biomarkers in Medical Images
This project aims to develop novel methods for discovering and visualising optimal biomarkers from chest computed tomography images based on extensions of recently developed deep reinforcement learning techniques. The extensions proposed in this project will advance medical image analysis by allowing an efficient analysis of large dimensionality inputs in their original high resolution. In addition, this project will be the first approach capable of discovering previously unknown biomarkers associated with important clinical outcomes. The project will validate the approach on a real-world case study data set concerning the prediction of five-year survival of chronic disease.
A/Prof Gustavo Carneiro; Prof Andrew Bradley; Prof Lyle Palmer; Dr Jacinto Nascimento
ARC Grant ID: DP180103232
Learning the deep structure of Images
This project seeks to develop technologies that will help computer vision interpret the whole visible scene, rather than just some of the objects therein. Existing automated methods for understanding images perform well at recognising specific objects in canonical poses, but the problem of whole image interpretation is far more challenging. Convolutional neural networks (CNN) have underpinned recent progress in object recognition, but whole-image understanding cannot be tackled similarly because the number of possible combinations of objects is too large. The project thus proposes a graph-based generalisation of the CNN approach which allows scene structure to be learned explicitly. This would represent an important step towards providing computers with robust vision, allowing them to interact with their environment.
Professor Anton van den Hengel; Dr Anthony Dick; Dr Lingqiao Liu
ARC Grant ID: DP160103710
Lifelong Computer Vision Systems
The aim of the project is to develop robust computer vision systems that can operate over a wide area and over long periods. This is a major challenge because the geometry and appearance of an environment can change over time, and long-term operation requires robustness to this change. The outcome will be a system that can capture an understanding of a wide area in real time, through building a geometric map endowed with semantic descriptions, and which uses machine learning to continuously improve performance. The significance will lie in turning an inexpensive camera into a high-level sensor of the world, ushering in cognitive robotics and autonomous systems.
ARC Grant ID: FL1300100102
Semantic change detection through large scale learning
Identifying whether there has been a significant change in a scene from a set of images is an important practical task, and has received much attention. The problem has been, however, that although existing statistical techniques perform reasonably well, it has been impossible to achieve the high levels of accuracy demanded by most real applications. This is due to the fact that changes in pixel intensity are not a particularly good indicator of significant change in a scene. We propose a semantic change detection approach which aims to classify the content of an image before attempting to identify change. This technology builds upon recent developments in large-scale classification which have dramatically improved both accuracy and speed.
Professor Anton van den Hengel; Professor Chunhua Shen; Dr Anders Eriksson; Dr Qinfeng Shi; BAE Systems
ARC Grant ID: LP130100156
Continuously learning to see
The ultimate goal of computer vision is to make a machine able to understand the world through analysis of images or videos. The new machine learning techniques developed in this project will enable previously impossible methods of computer vision and help strengthen Australian competitiveness in this important area.
ARC Grant ID: FT120100969
Image search for simulator content creation
3D content creation represents one of the most labour intensive stages of the process of constructing virtual environments such as simulators and games. In many cases it is possible to capture images or video of the environment to be simulated which may be used to assist the modelling process. This project aims to develop technologies based on search by which such imagery may provide both shape and semantic information to assist in the modelling process. The project builds upon recent developments in bag-of-words methods for image search. Particularly, we propose a novel method by which information latent in the image database may be identified and used to improve generative model underpinning this type of image search.
Anton van den Hengel, Anthony Dick, Sydac
ARC Grant ID: LP100100791