API Reference
AstroLink
- class astrolink.astrolink.AstroLink(P, d_intrinsic=None, weights=None, k_den=20, adaptive=1, S='auto', k_link='auto', h_style=1, workers=8, verbose=0, working_memory=None)
A class to represent the AstroLink clustering algorithm.
AstroLink, the algorithmic successor of CluSTAR-ND and Halo-OPTICS, finds clusters from point-cloud data that are statistical outliers from the stochastic clumps within the input data. In addition to this, clusters are defined as contiguous groupings of points with local density greater than some bounding iso-density contour.
Given a point-cloud data set, AstroLink; * (optionally) transforms the data so that each feature has unit variance, * estimates the local-density of each point, * aggregates points into an ordered list while keeping track of groups, * calculates the prominence (a measure of clusteredness) for each group, * fits a descriptive model to the prominence distribution, and * labels outliers (from this distribution) as clusters.
Parameters
- Pnumpy.ndarray of shape (n_samples, d_features)
A feature array of the point-cloud data set from which AstroLink finds clusters.
- d_intrinsicint, default = None
The intrinsic dimensionality of the input data. If d_intrinsic is set to None, then AstroLink assumes that the data has the same dimensionality as the number of features in P.
- weightsnumpy.ndarray of shape (n_samples,), default = None
The weights of each point in the input data. If weights is set to None, then all points are given equal weight. Otherwise, the weights must be a 1D numpy array with the same length as the number of samples in P.
- k_denint, default = 20
The number of nearest neighbours used to estimate the local density of each point.
- adaptiveint, default = 1
A flag that determines the behaviour of the input data transformation. If adaptive is set to 0, then AstroLink does not transform the data. If adaptive is set to 1, then AstroLink rescales each feature so that is has unit variance.
- Sfloat or str, default = ‘auto’
The lower significance threshold of any clusters found, i.e clusters are S-sigma outliers from the stochastic clumps within the input data. If S is set to a float, then this value is used directly. If S is set to ‘auto’, then a data-driven value of S is used.
- k_linkint or str, default = ‘auto’
The number of nearest neighbours used to aggregate points. If k_link is set to an int, this value is used directly. If k_link is set to ‘auto’, then a data-driven value of k_link is used.
- h_styleint, default = 1
A flag that determines the behaviour of the cluster hierarchy. If h_style is set to 0, then the resultant hierarchy is styled similarly to SubFind and EnLink. If h_style is set to 1, then additional clusters are incorporated into the cluster hierarchy.
- workersint, default = 8
The number of processors used in parallelised computations. If workers` is set to -1, then AstroLink will use all processors available. Otherwise, workers must be a value between 1 and N_cpu.
- verboseint, default = 0
The verbosity of the AstroLink class. If verbose is set to 0, then AstroLink will not report any of its activity. Increasing verbose will make AstroLink report more of its activity.
- working_memoryfloat or int, default = None
The amount of working memory (in bytes) to use for computations. If working_memory is set to None, then this will be set to half of the available memory.
Attributes
- logRhonumpy.ndarray of shape (n_samples,)
The normalised log-scaled local densities of each point, such that logRho[i] corresponds to the data point P[i].
- orderingnumpy.ndarray of shape (n_samples,)
The ordered list of points, such that P[ordering[i]] is the point that belongs in the i-th position in the ordered list. The ordered-density plot, which can used to visualise clustering structure, can be created by plotting y = logRho[ordering] vs x = range(n_samples).
- clustersnumpy.ndarray of shape (n_clusters, 2)
The start and end positions of each cluster as it appears in the ordered list, such that ordering[clusters[i, 0]:clusters[i, 1]] gives an array of the indices of the points within cluster i.
- idsnumpy.ndarray of shape (n_clusters,)
The id strings of the clusters, such that ids[i] corresponds to cluster i. These id strings have a hierarchical structure to their notation; the cluster with id ‘1’, is the parent of the cluster with id ‘1-1’, which is the sibling of the cluster with id ‘1-2’, which is the parent of the cluster with id ‘1-2-1’ (and so on…).
- significancesnumpy.ndarray of shape (n_clusters,)
The statistical significance of each cluster, such that significance[i] corresponds to cluster i. These values equivalent to ‘how many sigma’ outliers the clusters are from the noise within the input data.
- groupsnumpy.ndarray of shape (n_groups, 3)
Similar to clusters, however groups contains the start and end positions of all pairs of larger and smaller groups that were merged together during the aggregation process. Here groups[:, 0] and groups[:, 1] correspond to the start and end positions of the larger groups, while groups[:, 1] and groups[:, 2] correspond to the start and end positions of the smaller groups. groups can be used to visualise the aggregation merge tree.
- prominencesnumpy.ndarray of shape (n_groups, 2)
The prominence values of each group in groups, such that prominences[i] contains the prominence values of both groups stored in groups[i].
- groups_sigsnumpy.ndarray of shape (n_groups, 2)
The statistical significance of each group, such that groups_sigs[i] contains the statistical significance values of both groups stored in groups[i].
- pFitnumpy.ndarray of shape (2,)
The shape parameters, [a, b], of the Beta distribution model that fits the prominences.
- aggregate()
Aggregates the data points together to form the ordered list whilst keeping track of groups.
Computes edge weights between each point and each of its k_link nearest neighbours, sorts these into descending order, aggregates points in this order while keeping track of structural information about the data.
This method requires the logRho and ‘kNN’ attributes to have already been created, via the estimate_density_and_kNN method or otherwise.
This method generates the ordering, groups, prominences attributes.
This method deletes the kNN attribute.
- compute_significances()
Computes statistical significances for all groups by fitting a descriptive model to their prominences.
Fits the prominence model (a Beta distribution) by minimising the Wasserstein-1 distance between the numerical and empirical cdfs. The noise model (with model-fitted parameters) is then used alongside the standard normal distribution to transform prominence values into statistical significance values.
The method requires the prominences attribute to have already been created, via the aggregate method or otherwise.
This method generates the groups_sigs and pFit attributes.
- estimate_density_and_kNN()
Estimates the normalised log-scaled local density of each point from its k_den nearest neighbours and then keeps only the k_link nearest neighbours of each point.
Uses a KD-Tree to find the k_den nearest neighbours of every point in the input data and then uses these neighbours to estimate the local density of each point using an Epanechnikov kernel + Balloon estimator. The local densities are then log-scaled and normalised between 0 and 1.
This method requires the P_transform attribute to have already been created, via the transform_data() method or otherwise.
This method generates the logRho and kNN attributes.
This method deletes the P_transform attribute.
- extract_clusters()
Classifies groups that have significance of at least S as clusters and forms the hierarchy according to the parameter h_style.
First classifies any groups that are statistical outliers. Then if h_style is set to 1, finds the corresponding groups that are also statistical outliers. For each of these that are the smallest out of those that share the same starting position within the ordered list, classify them too as clusters. The input data is always classified as the root cluster in addition to these overdensities. Finally, generate the array of id strings for the clusters.
This method requires the groups attribute to have already been created, via the aggregate() method or otherwise. It also requires the groups_sigs and pFit attributes to have already been created, via the compute_significance method or otherwise.
This method generates the ‘clusters’, ids, and significances attributes.
- run()
Runs the AstroLink algorithm and produces a hierarchy of clusters from the input data.
This method runs transform_data(), estimate_density_and_kNN(), aggregate(), compute_significances(), extract_clusters().
- transform_data()
Transforms the input data according to the parameter adaptive.
If adaptive is set to 0, then the data is not transformed. However, if adaptive is set to 1 (default), then each feature of the data is rescaled so that it has unit variance.
This method generates the P_transform attribute.
Input/Output
Loading and saving routines to be used with the AstroLink class.
Author: William H. Oliver <william.hardie.oliver@gmail.com> License: MIT
Visualization
Plotting routines to be used with the AstroLink class.
Author: William H. Oliver <william.hardie.oliver@gmail.com> License: MIT
- astrolink.visualize.aggregationTree(clusterer, ax=None, plotKwargs={}, treeKwargs={})
Make the AstroLink aggregation tree plot.
Parameters
- clustererAstroLink
An instance of the AstroLink class.
- axmatplotlib.axes._axes.Axes
The axes on which to plot the aggregation tree. If not provided, the current axes will be used.
- plotKwargsdict, optional
Keyword arguments to be passed to the plot function that makes the ordered-density function.
- treeKwargsdict, optional
Keyword arguments to be passed to the hlines and vlines functions that make the aggregation tree.
- astrolink.visualize.labelsOnX(clusterer, X, skipZeroth=True, ax=None, scatterKwargs={})
Make a plot of the data X coloured by the AstroLink cluster labels.
Parameters
- clustererAstroLink
An instance of the AstroLink class.
- Xarray
The data to be plotted.
- axmatplotlib.axes._axes.Axes
The axes on which to plot the data. If not provided, the current axes will be used.
- scatterKwargsdict, optional
Keyword arguments to be passed to the scatter function.
- astrolink.visualize.logRhoOnX(clusterer, X, ax=None, colorbar=True, scatterKwargs={}, colorbarKwargs={})
Make a plot of the data X coloured by the AstroLink log-density on the data.
Parameters
- clustererAstroLink
An instance of the AstroLink class.
- Xarray
The data to be plotted.
- axmatplotlib.axes._axes.Axes
The axes on which to plot the data. If not provided, the current axes will be used.
- colorbarbool, optional
Whether to add a colour bar to the plot.
- scatterKwargsdict, optional
Keyword arguments to be passed to the scatter function.
- colorbarKwargsdict, optional
Keyword arguments to be passed to the colorbar function.
- astrolink.visualize.orderedDensity(clusterer, skipZeroth=True, ax=None, plotKwargs={}, fillKwargs={})
Make the AstroLink ordered-density plot.
Parameters
- clustererAstroLink
An instance of the AstroLink class.
- axmatplotlib.axes._axes.Axes, optional
The axes on which to plot the ordered density. If not provided, the current axes will be used.
- plotKwargsdict, optional
Keyword arguments to be passed to the plot function that makes the ordered-density function.
- fillKwargsdict, optional
Keyword arguments to be passed to the fill_between function that highlights the clusters.
Returns
- linematplotlib.lines.Line2D
The line object representing the dendrogram.
- polyCollectionslist of matplotlib.collections.PolyCollection
The PolyCollection objects representing the clusters as filled regions on the ordered-density plot.
- astrolink.visualize.prominenceModel(clusterer, ax=None, histKwargs={}, modelKwargs={}, cutoffKwargs={})
Make a plot of AstroLink prominences.
Parameters
- clustererAstroLink
An instance of the AstroLink class.
- axmatplotlib.axes._axes.Axes
The axes on which to plot the prominences. If not provided, the current axes will be used.
- histKwargsdict, optional
Keyword arguments to be passed to the hist function.
- modelKwargsdict, optional
Keyword arguments to be passed to the plot function for the fitted prominence model.
- cutoffKwargsdict, optional
Keyword arguments to be passed to the vlines function for the cutoff value.
Returns
- harray
The values of the histogram bins.
- binsarray
The edges of the bins.
- patcheslist of matplotlib.patches.Patch
The Patch objects representing the histogram bars.
- linematplotlib.lines.Line2D
The line object representing the fitted prominence model.
- lineCollectionmatplotlib.collections.LineCollection
The LineCollection object representing the cutoff over the prominences histogram.
- astrolink.visualize.significanceModel(clusterer, ax=None, histKwargs={}, modelKwargs={}, cutoffKwargs={})
Make a plot of AstroLink significances.
Parameters
- clustererAstroLink
An instance of the AstroLink class.
- axmatplotlib.axes._axes.Axes
The axes on which to plot the significances. If not provided, the current axes will be used.
- histKwargsdict, optional
Keyword arguments to be passed to the hist function.
- modelKwargsdict, optional
Keyword arguments to be passed to the plot function for the fitted significance model.
- cutoffKwargsdict, optional
Keyword arguments to be passed to the vlines function for the cutoff value.
Returns
- harray
The values of the histogram bins.
- binsarray
The edges of the bins.
- patcheslist of matplotlib.patches.Patch
The Patch objects representing the histogram bars.
- linematplotlib.lines.Line2D
The line object representing the fitted prominence model.
- lineCollectionmatplotlib.collections.LineCollection
The LineCollection object representing the cutoff over the prominences histogram.