Rich Apodaca recently blogged about this stereochemistry related series of posts. He correctly pointed out that the automorphism group lies at the heart of the Razinger paper on stereoismer generation. However, before the automorphism group can be used, all stereocenters have to be found.
In a
previous post I briefly described rule 1 and gave some examples. Below are more examples of these para-stereocenters identified by applying rule 1. As you will notice, most of these structures contain no stereocenter and are used to avoid false positives in the unit tests.

While rule 1 handles interdependant stereocenters for cycles, both rule 2 and rule 3 handle individual stereocenters and can be applied directly (rule 1 is applied recursively). The rules can schematically be summarized as:
2) tetracoordinate carbon is para-stereocenter if it attached:
2a) 1 or 2 pairs of identical symmetry classes (e.g. 1123, 1122), each pair contains at least :
2a') >= 1 true-stereocenter (have 4 different symmetry classes) -- or --
2a") >= 2 para-stereocenters (from rule 1)
2b) 3 or 4 identical symmetry classes (e.g. 1112, 1111), duplicated symmetry class ligand contains:
2b') >= 2 true-stereocenters -- or --
2b") >= 2 separate assemblies (merged rings) with each >= 2 para-stereocenters
3) double bond is para-stereocenter if terminal with identical symmetry classes contains at least:
3') >= 1 true-stereocenter
3") >= 2 para-stereocenters
Some examples of rule 2 & 3:

Now that all stereocenters are found, a (very) short introduction to permutations and permutation groups will help us get to the automorphism group (This can be skipped if you are familiar with it). A permutations is simply a reordering of a set of numbers. Although there are several ways to represent permutations, the matrix representation is most useful here. A permutation group is unsurprisingly a group or collection of Permutations. (see Razinger paper + wikipedia
Permutation Group &
Permutation Matrix)
The automorphism group is a subgroup of the Sn group containing all permutations. Since elements are not allowed to repeat, there will be n! permutations in Sn where n is the number of atoms. In general, the term "auto" means self and permutations belonging to the automorphism group change the atom labeling in such a way that the molecule isn't changed (i.e. no bonds broken or neighbors changed, the molecule remains itself). Mathematically this can be expressed using the adjacency matrix (A) and the permutation matrix (P) in the equation:

While a brute force check of all n! permutations would work for small molecules, it is not efficient enough for use with most drug-like molecules. To limit the number of automorphism candidates, symmetry classes can be used to provide an initial partitioning. Atoms with identical symmetry class are said to be in the same orbit. Only permutations within orbits are allowed (i.e. to be part automorphism group). These intra-orbit permutations can easily be computed in c++ using std::
next_permutation. These intra-orbit permutations are all candidates for the automorphism group (G) and will be added to G if the equation holds (duplicates are ignored, especially the identity permutation will be duplicated many times).
Next, all inter-orbit permutations still need to be considered. This is achieved by "multiplying" the permutations (I used matrix multiplication but other methods are possible) and this gives rise to a new set of candidates which are also checked using the equation. All permutations in G now make up the automorphism group.
Below are 3 examples to illustrate how these "obrits" limit the number of permutations. The structure on the left, has 2 orbits, each containing 2 atoms. This structures' Sn group has 7! or 5040 permutations. Using the orbits we can reduce the number of candidates to 2! x 2! = 4. The structure in the middle also has 2 orbits but one orbit has 2 atoms and the other has 4 (6! = 720 -> 2! x 4! = 48). The structure on the right has 3 orbits, each containing 2 atoms (9! = 362880 -> 2! x 2! x 2! = 8).

While OpenBabel doesn't have code to compute the automorphism group yet,
here is some proof of concept code which will be integrated later. It also contains a method to compute all permutations of the Sn group which can be used in combination with small molecules to validate the automorphism group generating algorithm.
Once the stereocenters are identified and the automorphism group has been found, this information can be used to eventually create a stereoparity matrix. The details and code will follow in a later post but knowing that duplicate rows in this matrix represent the same stereoisomer should be motivating :-) (for both stereoisomer generation & canonicalization)