Research Project (1992)

Recognition of partially occluded objects in cluttered scenes

Laurenz Wiskott and Christoph von der Malsburg

This project deals with the recognition of toy objects which are arranged in scenes, such that they occlude each other up to 70%; see the figure (left).

cluttered scene ... (31 kB) ... and its interpretation (7 kB)

Figure: A scene with five objects occluding each other. Left: The original image. Superimposed are the graphs of the basket and the nutcracker. Nodes considered visible are black, those considered occluded are white. Right: The scene is interpreted by the system under the assumption that all objects in the scene are known.

Each toy object is represented by a labeled graph. The nodes are additionally labeled with binary variables indicating whether a node is considered to be visible or occluded. Here it is particularly advantageous that the graph structure permits refering to local subregions of the object. For finding the most promising locations for each model graph, we apply our standard elastic graph matching without local distortions. The goal of the system is then to find for each graph coherent regions of high similarity, considered as being visible, and other regions of low similarity, considered as being occluded. Based on size and average similarity of the visible region it is decided whether the object is present in the scene or not. If all objects in the scene are known (but it is not known which objects are in the scene), the system can take additional information from the constraints that objects have an order in depth and that at each image location only one object may be visible.

We have tested the system with a gallery of 13 toy objects on 30 scenes with 3 to 6 objects each and a total of 121 objects to recognize. The system recognized 97% of the objects correctly, and it determined all but three depth relations successfully.

Relevant Publications:

Black colored reference are the principal ones. Gray colored references are listed for the sake of completeness only. They contain little additional information. .ps-files are optimized for printing; .pdf-files are optimized for viewing at the computer.

  1. Wiskott, L. (1995).
    Labeled graphs and dynamic link matching for face recognition and scene analysis.
    Verlag Harri Deutsch, Thun - Frankfurt am Main, Reihe Physik 53 (PhD thesis), ISBN 3-8171-1481-8, 110 pages.
    (bibtex, abstract.html, paper.pdf,

  2. Wiskott, L. and von der Malsburg, C. (1994).
    A neural system for the recognition of partially occluded objects in cluttered scenes.
    In Advances in Pattern Recognition Systems using Neural Networks Technologies, eds. I. Guyon, and P.S.P. Wang, vol. 7 in series Machine Perception and Artificial Intelligence, publ. World Scientific, Singapore, pp. 291-304 (a reprint of WisMal93).
    (bibtex, abstract.html, paper.pdf,,

  3. Doursat, R., Konen, W., Lades, M., von der Malsburg, C., Vorbrüggen, J.C., Wiskott, L., and Würtz, R.P. (1993).
    Neural mechanisms of elastic pattern matching.
    Internal report, IR-INI 93-01, Institut für Neuroinformatik, Ruhr-Universität Bochum, 44780 Bochum, Germany, 13 pages.
    (bibtex, abstract.html,

  4. Wiskott, L. and von der Malsburg, C. (1993).
    A neural system for the recognition of partially occluded objects in cluttered scenes.
    Int'l J. of Pattern Recognition and Artificial Intelligence, 7(4):935-948.
    (bibtex, abstract.html, paper.pdf,,

Related Project:

setup April, 1996; updated December 14, 1998
Laurenz Wiskott,