J. Sinkkonen and S. Kaski

We have extended information-bottleneck type clustering to continuous
multivariate co-occurrence data (x,y). Instead of pairs of discrete
data, for instance words and documents, x and y are vectors, for
instance expression and transcription binding profiles of genes.  The
margin clusters for x and y are defined as Voronoi partitionings, but
they are optimized by a distributional cost (mutual information)
instead of the standard K-means quantization error.  Either one or two
margins can be clustered, corresponding to the classical and symmetric
information bottleneck. Alternative cost functions can be derived from
measures of dependency used in the context of contingency tables.
These are also applicable to the classic IB with discrete margins.
Results are promising for small data sets, while still being
asymptotically equivalent to mutual information for large data sets.