Clustering Methods

Clustering turns a projected map of points into groups that can be labeled, summarized, colored, selected, and navigated. A cluster is not a second copy of the source data. It is a hierarchy node over points in one map.

Mantis currently registers three clustering strategies:

• k_means: agglomerative hierarchical linkage through fastcluster.
• h_dbscan: density-based hierarchical clustering through HDBSCAN.
• leiden: community detection over a nearest-neighbor graph.

The default is k_means with Ward linkage and Euclidean distance. Despite its configuration name, this path does not run scikit-learn KMeans.

Pipeline

Clustering runs after projection has produced point embeddings. The backend:

1. loads the map, points, field types, and clustering_config;
2. selects the configured clustering space and strategy;
3. builds a hierarchy or community tree;
4. selects a usable depth and prunes the tree;
5. reassigns each point to its nearest surviving ancestor;
6. writes Cluster rows and updates each point’s Idea.cluster relation; and
7. computes cluster centers, colors, labels, summaries, and statistics in later synthesis stages.

Maps with coordinate fields always cluster in visual space. This keeps groups aligned with the supplied coordinates instead of switching to another embedding.

Clustering spaces

clustering_space chooses which vector is passed to the strategy:

Value	Point field	Meaning
visual	embedding	The displayed 2D map projection.
low_dim	low_dim_embedding	An intermediate reduced representation.
high_dim	raw_embedding	The original high-dimensional embedding.

The task fails explicitly when the selected field is missing or null for every point. Trustworthiness-based depth selection is unavailable for high_dim clustering because it needs a separate high-dimensional reference hierarchy.

Strategy behavior

k_means

The historical k_means name is a compatibility label. The implementation calls fastcluster.linkage, converts the linkage matrix to a SciPy tree, and records each internal node as a Mantis cluster.

Defaults:

• linkage_method: ward
• metric: euclidean
• min_cluster_size: 5

The configuration also accepts complete, average, and single linkage and either Euclidean or cosine distance where the selected linkage supports it. n_clusters remains part of the configuration contract but is not used by the current hierarchy builder.

This strategy produces a deep binary hierarchy that follows pairwise distance. It is useful when broad-to-specific navigation is more important than detecting irregular density islands.

fastcluster.linkage has quadratic memory growth. When a map configured for k_means contains more than 100,000 points, Mantis automatically switches the job to h_dbscan to avoid an out-of-memory failure.

h_dbscan

HDBSCAN finds dense regions and exposes its single-linkage tree. Mantis converts that tree into the same parent/depth format used by the rest of the pipeline.

Defaults:

• min_cluster_size: 5
• metric: euclidean
• cluster_selection_method: eom
• algorithm: best

cluster_selection_method may also be leaf. HDBSCAN is a better fit when cluster density varies or groups have irregular shapes. Its hierarchy may be less balanced than the linkage strategy. Points classified as noise are still assigned within the generated Mantis hierarchy rather than left without a cluster relation.

leiden

Leiden builds a symmetrized k-nearest-neighbor graph from the selected embedding and detects graph communities with the Leiden algorithm.

Defaults:

• n_neighbors: 15
• resolution: 1.0
• n_iterations: 2
• metric: euclidean

The output is a flat two-level hierarchy: one root at depth 0 and one child per community at depth 1. Higher resolution generally yields more communities. Leiden is useful when graph connectivity is the main signal, but it does not provide the multi-level drill-down produced by the other two strategies.

Depth selection

Hierarchical strategies can create more levels than the interface should expose. optimal_depth_detection_strategy chooses how Mantis prunes them.

sse

This is the default. Mantis evaluates depths up to max_depth (default 10), reassigns points at each candidate depth, and computes within-cluster sum of squared errors from the 2D embedding. It keeps the depth with the lowest SSE and stops early after SSE worsens for three consecutive depths.

trustworthiness

Mantis builds a second hierarchy from raw_embedding, compares the low- and high-dimensional assignments at each depth using Adjusted Rand Index, and keeps the best agreement. The search stops after the score drops for three consecutive depths.

This option asks whether visible groups preserve high-dimensional relationships; it does not calculate scikit-learn’s point-neighborhood trustworthiness metric.

fixed

The fixed strategy derives one depth from map size:

max(1, floor(log2(point_count)) - 2)

It then prunes directly to that depth. This is deterministic and inexpensive, but it does not compare cluster quality across depths.

Leiden already emits only depths 0 and 1, so depth pruning cannot create a deeper community hierarchy.

Stored objects

Each backend Cluster belongs to one Map and stores:

• id and map;
• parent and depth;
• label, summary, and keywords;
• color; and
• a 2D center.

Each point is an Idea; its cluster relation points to its assigned surviving cluster. Parent relations let callers climb from a specific group to broader groups without duplicating point records.

The frontend Cluster adds delivery state:

• dataPoints contains points currently visible in the loaded level-of-detail sample;
• allDataPoints may contain all delivered descendants and can be derived in the browser from leaf assignments plus parent edges; and
• pointCount is the authoritative source-map count, including points omitted from the current level-of-detail sample.

Use pointCount for sizes. Do not treat dataPoints.length or allDataPoints.length as the total population on a partially delivered map.

Product behavior

Clusters support more than map coloring:

• labels and centers make neighborhoods readable on the map;
• the Tree panel exposes broad-to-specific navigation;
• selections, bags, and overlays can group points or climb cluster ancestors;
• summaries and keywords provide compact context to panels and agents; and
• cluster counts allow large maps to remain useful before every point is loaded.

Changing the method can change the interpretation of the same map. Linkage emphasizes nested distance relationships, HDBSCAN emphasizes density, and Leiden emphasizes graph communities. A visually appealing partition is not by itself evidence that the groups are semantically correct.

Statistics

Mantis stores field statistics on Map.statistics and clustering diagnostics as ClusteringMetric rows. The clustering metrics are:

• davies_bouldin_score;
• silhouette_score; and
• calinski_harabasz_score.

They are computed from the 2D embedding and assigned cluster labels. Silhouette may use a 10,000-point sample on larger maps. These scores describe compactness and separation; they do not measure label quality or semantic usefulness.

Source landmarks

The current implementation is split across the frontend and backend repositories:

• MantisAPI/src/synthesis/tasks/clustering.py
• MantisAPI/src/synthesis/tasks/clustering_utilities/k_means.py
• MantisAPI/src/synthesis/tasks/clustering_utilities/h_dbscan.py
• MantisAPI/src/synthesis/tasks/clustering_utilities/leiden.py
• MantisAPI/src/synthesis/tasks/clustering_utilities/depth_detection/
• MantisAPI/src/synthesis/tasks/cluster_metadata.py
• MantisAPI/src/synthesis/tasks/cluster_labeling.py
• MantisAPI/src/synthesis/tasks/statistics_computation_task.py
• MantisAPI/src/db/models.py
• Mantis/src/app/types.ts
• Mantis/src/app/components/data_provider/clusterMembership.ts