VV: DBSCAN Full V&V Complete

src/Plugins/SimplnxCore/docs/DBSCANFilter.md

@@ -35,7 +35,6 @@ An advantage of DBSCAN over other clustering approaches (e.g., k-means) is that
 All the available examples are 2D as they come from [Sci-Kit Learn Toy Datasets](https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html).
 
 Keeping in mind 0 is unlabeled, here is a table of the results:
-*Note: at the time of image capture a bug was showing the yellow as NaNs, but they were labeled with 3 in the cluster array.*
 
 | name            | Image                                                                          |
 |-----------------|--------------------------------------------------------------------------------|
@@ -92,6 +91,17 @@ Here are some additional visualization tips that make it easier to analyze the d
 - For datasets with less than 32 clusters, in the points view settings window, enabling `Interpret Values as Categories` is a great color scheme that shows clear distinctions between clusters
 - In the points view settings window, enabling `Color Legend` under `Annotations` helps distinguish clusters and process order.
 
+### Known Differences from Traditional DBSCAN
+
+This filter implements GDCF (Grid-based DBSCAN), which differs from the classic point-by-point DBSCAN algorithm in one important way: **core-object definition is at the grid-cell level, not the point level**.
+
+- **Traditional DBSCAN**: a data point `p` is a *core point* if its ε-ball contains ≥ `Minimum Points` data points (inclusive of `p` itself).
+- **This filter (GDCF)**: a *core grid* is a grid cell (side length = `Epsilon / sqrt(Dimensions)`) that contains ≥ `Minimum Points` data points.
+
+The consequence is that **very sparse micro-clusters may be classified as noise** in this filter even though traditional DBSCAN would find them. Specifically, if a small group of points individually have neighbors within ε but those neighbors span two adjacent grid cells (each with fewer than `Minimum Points` points), neither cell qualifies as a core grid. Traditional DBSCAN would still cluster these points; GDCF does not.
+
+For dense, well-clustered data this difference is imperceptible. For data with significant variation in local density — particularly small clusters of 2–4 points embedded in otherwise sparse regions — results may differ from traditional DBSCAN implementations (e.g., scikit-learn's `DBSCAN`). In such cases, lowering `Minimum Points` (e.g., from 3 to 2 or 1) or increasing `Epsilon` slightly may recover the expected clusters.
+
 ### Hyperparameter Tuning
 
 This implementation of DBSCAN uses a grid approach to greatly increase the speed in which it is processed. This comes with a few caveats compared to the traditional algorithm, but in many ways it is easier to comprehend the effect of hyperparameter on the output. In this section we will be discussing just that, as well as how to optimize and quickly identify good initial guesses.
@@ -146,6 +156,12 @@ Additionally, oddities such as duplicates in the dataset or non-standardized dat
 **If your algorithm is excessively slow.**
 This can obviously be caused by large datasets, but it can be mitigated with some changes. Firstly, the "Parse Order" parameter can result in immediate speedups of 60% or more on a majority of datasets by switching to `Low Density First`. The idea being that lower density regions are cheaper for merge checks, so other denser core grids can be picked off early if they are close to sparser core grids, meaning that the expensive grids have less of a chance of running against one another. Another change is tightening the voxels grids by lowering the `Epsilon` and reducing the `Minimum Points` slightly. For ideal performance, in the vast majority of cases, you want to reach the lowest value for both of these that still produces expected clustering. This is because the most costly part is the distance check most of the time. Logically, grids with fewer points run less distance checks.
 
+**If your algorithm is consuming excessive memory.**
+Memory usage scales with the total number of grid cells — occupied and empty alike — proportional to `(bounding box per-axis range / cell side length)^D`, where cell side length is `Epsilon / sqrt(D)`. For 3D data this grows cubically. Two common causes:
+
+- **Small `Epsilon`**: reduces cell size, increasing total cell count across the bounding box.
+- **Extreme outlier points**: a single unmasked outlier far from the main dataset inflates the bounding box in one or more axes, multiplying total cell count. Consider masking or removing outliers before running the filter.
+
 **If your algorithm spends more time in "cluster expansion pass:" than "Identifying Qualifying Independent Clusters".** See output window.
 There are many datasets that this is normal in such as `No Structure` from **Examples**. This typically happens when `Minimum Points` is too high. This results in too few Core grids being identified. Since few clusters are able to be formed, most of the time is spent in iterative loops expanding the clusters rather than just preforming the early merges in the Core grid step.
 

src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp

@@ -165,6 +165,10 @@ class HyperGridBitMap3D : public HyperGridBitMap
       // Build a set of non-empty grids and temporarily store their positions
       {
         usize numTup = inputArray.getNumberOfTuples();
+        // grids and gridMap below are both sized by total cell count (occupied + empty):
+        // dims[0] * dims[1] * dims[2], where each dims[i] ~ (bounding box per-axis range) / (epsilon / sqrt(D)).
+        // grids is bit-packed (1 bit/cell); gridMap is usize/cell — both are live simultaneously.
+        // Small epsilon or active extreme outlier points on 3D data can make this allocation needlessly expensive.
         std::vector<bool> grids(std::accumulate(dims.cbegin(), dims.cend(), static_cast<usize>(1), std::multiplies<>()), false);
         // Find num grid cells
         for(usize tup = 0; tup < numTup; tup++)
@@ -324,6 +328,7 @@ class HyperGridBitMap2D : public HyperGridBitMap
   {
     ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger();
 
+    messageHelper.sendMessage(" - Determining bounds...");
     // Load array bounds
     std::array<float32, 4> bounds = {std::numeric_limits<float32>::quiet_NaN(), std::numeric_limits<float32>::quiet_NaN(), std::numeric_limits<float32>::quiet_NaN(),
                                      std::numeric_limits<float32>::quiet_NaN()};
@@ -372,6 +377,10 @@ class HyperGridBitMap2D : public HyperGridBitMap
       // Build a set of non-empty grids and temporarily store their positions
       {
         usize numTup = inputArray.getNumberOfTuples();
+        // grids and gridMap below are both sized by total cell count (occupied + empty):
+        // dims[0] * dims[1], where each dims[i] ~ (bounding box per-axis range) / (epsilon / sqrt(D)).
+        // grids is bit-packed (1 bit/cell); gridMap is usize/cell — both are live simultaneously.
+        // Small epsilon or active extreme outlier points on 3D data can make this allocation needlessly expensive.
         std::vector<bool> grids(std::accumulate(dims.cbegin(), dims.cend(), static_cast<usize>(1), std::multiplies<>()), false);
         // Find num grid cells
         for(usize tup = 0; tup < numTup; tup++)
@@ -637,12 +646,11 @@ struct ClusterForest
 
   usize findClusterRoot(usize gridId)
   {
-    if(clusterForestNodes[gridId].parent == gridId)
+    while(clusterForestNodes[gridId].parent != gridId)
     {
-      return gridId;
+      gridId = clusterForestNodes[gridId].parent;
     }
-
-    return findClusterRoot(clusterForestNodes[gridId].parent);
+    return gridId;
   }
 
   /**
@@ -747,31 +755,18 @@ class GDCF
       break;
     }
     case DBSCAN::ParseOrder::Random: {
-      std::mt19937_64 gen(seed);
-      std::uniform_real_distribution<float64> dist(0, 1);
-
-      auto maxIdx = static_cast<float64>(coreGridIds.size() - 1);
-
-      //--- Shuffle elements by randomly exchanging each with one other.
-      for(usize i = 1; i < coreGridIds.size(); i++)
-      {
-        auto r = static_cast<usize>(std::floor(dist(gen) * maxIdx)); // Random remaining position.
-
-        std::swap(coreGridIds[i], coreGridIds[r]);
-      }
-
-      break;
+      [[fallthrough]];
     }
     case DBSCAN::SeededRandom: {
       std::mt19937_64 gen(seed);
       std::uniform_real_distribution<float64> dist(0, 1);
 
-      auto maxIdx = static_cast<float64>(coreGridIds.size() - 1);
+      const auto maxIdx = static_cast<float64>(coreGridIds.size() - 1);
 
       //--- Shuffle elements by randomly exchanging each with one other.
       for(usize i = 1; i < coreGridIds.size(); i++)
       {
-        auto r = static_cast<usize>(std::floor(dist(gen) * maxIdx)); // Random remaining position.
+        const auto r = static_cast<usize>(std::floor(dist(gen) * maxIdx)); // Random remaining position.
 
         std::swap(coreGridIds[i], coreGridIds[r]);
       }
@@ -970,7 +965,8 @@ class GDCF
   const std::atomic_bool& m_ShouldCancel;
   MessageHelper& m_MessageHelper;
 
-  // Uses Hoare's method for speed
+  // First-element pivot quicksort partition (two-pointer, Hoare-style).
+  // Worst case O(n^2) if occupancy values are already sorted ascending — unlikely on real spatial data.
   usize ProcessSection(std::vector<usize>& sorted, usize begin, usize end) const
   {
     const usize threshold = hyperGridBitMap.gridVoxels[sorted[begin]].size();
@@ -1019,6 +1015,10 @@ class GDCF
   {
     for(usize pPointId : hyperGridBitMap.gridVoxels[pGridId])
     {
+      if(m_ShouldCancel)
+      {
+        return false;
+      }
       for(usize qPointId : hyperGridBitMap.gridVoxels[qGridId])
       {
         float64 dist = ClusterUtilities::GetDistance(m_InputDataStore, (HGBPT::Dimensions * pPointId), m_InputDataStore, (HGBPT::Dimensions * qPointId), HGBPT::Dimensions, m_DistMetric);
@@ -1074,16 +1074,12 @@ struct DBSCANFunctor
     {
       return RunAlgorithm<GDCF<HyperGridBitMap2D, T>, T>(inputValues, inputArray, mask, featureIds, messageHelper, shouldCancel);
     }
-    else if(inputArray.getNumberOfComponents() == 3)
+    if(inputArray.getNumberOfComponents() == 3)
     {
       return RunAlgorithm<GDCF<HyperGridBitMap3D, T>, T>(inputValues, inputArray, mask, featureIds, messageHelper, shouldCancel);
     }
-    else
-    {
-      return MakeErrorResult(-54060, fmt::format("Input array has {} components but only 2 or 3 are accepted.", inputArray.getNumberOfComponents()));
-    }
 
-    return {};
+    return MakeErrorResult(-54060, fmt::format("Input array has {} components but only 2 or 3 are accepted.", inputArray.getNumberOfComponents()));
   }
 };
 } // namespace
@@ -1133,7 +1129,7 @@ Result<> DBSCAN::operator()()
 
   messageHelper.sendMessage("Resizing clustering Attribute Matrix:");
   auto& featureIdsDataStore = featureIds.getDataStoreRef();
-  int32 maxCluster = *std::max_element(featureIdsDataStore.begin(), featureIdsDataStore.end());
+  const int32 maxCluster = *std::max_element(featureIdsDataStore.begin(), featureIdsDataStore.end());
   m_DataStructure.getDataAs<AttributeMatrix>(m_InputValues->FeatureAM)->resizeTuples(ShapeType{static_cast<usize>(maxCluster + 1)});
 
   return result;

src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp

@@ -90,7 +90,7 @@ Parameters DBSCANFilter::parameters() const
                                                           ArraySelectionParameter::AllowedTypes{DataType::boolean, DataType::uint8}));
 
   params.insertSeparator(Parameters::Separator{"Input Data Objects"});
-  params.insert(std::make_unique<ArraySelectionParameter>(k_SelectedArrayPath_Key, "Attribute Array to Cluster", "The data array to cluster", DataPath{}, nx::core::GetAllNumericTypes(),
+  params.insert(std::make_unique<ArraySelectionParameter>(k_SelectedArrayPath_Key, "Attribute Array to Cluster", "The data array to cluster", DataPath{}, GetAllNumericTypes(),
                                                           ArraySelectionParameter::AllowedComponentShapes{{2}, {3}}));
 
   params.insertSeparator(Parameters::Separator{"Output Data Object(s)"});
@@ -133,13 +133,13 @@ IFilter::UniquePointer DBSCANFilter::clone() const
 IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataStructure, const Arguments& filterArgs, const MessageHandler& messageHandler, const std::atomic_bool& shouldCancel,
                                                      const ExecutionContext& executionContext) const
 {
-  auto pUseMaskValue = filterArgs.value<bool>(k_UseMask_Key);
-  auto pEpsilonValue = filterArgs.value<float32>(k_Epsilon_Key);
-  auto pMinPointsValue = filterArgs.value<int32>(k_MinPoints_Key);
-  auto pSelectedArrayPathValue = filterArgs.value<DataPath>(k_SelectedArrayPath_Key);
-  auto pMaskArrayPathValue = filterArgs.value<DataPath>(k_MaskArrayPath_Key);
-  auto pFeatureIdsArrayNameValue = filterArgs.value<std::string>(k_FeatureIdsArrayName_Key);
-  auto pFeatureAMPathValue = filterArgs.value<DataPath>(k_FeatureAMPath_Key);
+  const auto pUseMaskValue = filterArgs.value<bool>(k_UseMask_Key);
+  const auto pEpsilonValue = filterArgs.value<float32>(k_Epsilon_Key);
+  const auto pMinPointsValue = filterArgs.value<int32>(k_MinPoints_Key);
+  const auto pSelectedArrayPathValue = filterArgs.value<DataPath>(k_SelectedArrayPath_Key);
+  const auto pMaskArrayPathValue = filterArgs.value<DataPath>(k_MaskArrayPath_Key);
+  const auto pFeatureIdsArrayNameValue = filterArgs.value<std::string>(k_FeatureIdsArrayName_Key);
+  const auto pFeatureAMPathValue = filterArgs.value<DataPath>(k_FeatureAMPath_Key);
 
   if(pEpsilonValue <= 0.0f)
   {
@@ -150,17 +150,11 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt
     return MakePreflightErrorResult(-7585, fmt::format("Minimum Points value {} must be greater than 0.", pMinPointsValue));
   }
 
-  nx::core::Result<OutputActions> resultOutputActions;
-  std::vector<PreflightValue> preflightUpdatedValues;
+  Result<OutputActions> resultOutputActions;
 
-  auto clusterArray = dataStructure.getDataAs<IDataArray>(pSelectedArrayPathValue);
-  if(clusterArray == nullptr)
+  const auto& clusterArray = dataStructure.getDataRefAs<IDataArray>(pSelectedArrayPathValue);
   {
-    return MakePreflightErrorResult(-7586, "Array to Cluster MUST be a valid DataPath.");
-  }
-
-  {
-    auto createAction = std::make_unique<CreateArrayAction>(DataType::int32, clusterArray->getTupleShape(), std::vector<usize>{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue),
+    auto createAction = std::make_unique<CreateArrayAction>(DataType::int32, clusterArray.getTupleShape(), std::vector<usize>{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue),
                                                             CreateArrayAction::k_DefaultDataFormat, "0");
     resultOutputActions.value().appendAction(std::move(createAction));
   }
@@ -169,7 +163,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt
   {
     DataPath tempPath = DataPath({k_MaskName});
     {
-      auto createAction = std::make_unique<CreateArrayAction>(DataType::boolean, clusterArray->getTupleShape(), std::vector<usize>{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true");
+      auto createAction = std::make_unique<CreateArrayAction>(DataType::boolean, clusterArray.getTupleShape(), std::vector<usize>{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true");
       resultOutputActions.value().appendAction(std::move(createAction));
     }
 
@@ -190,7 +184,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt
   }
 
   // Return both the resultOutputActions and the preflightUpdatedValues via std::move()
-  return {std::move(resultOutputActions), std::move(preflightUpdatedValues)};
+  return {std::move(resultOutputActions)};
 }
 
 //------------------------------------------------------------------------------
@@ -225,7 +219,7 @@ Result<> DBSCANFilter::executeImpl(DataStructure& dataStructure, const Arguments
   inputValues.FeatureIdsArrayPath = inputValues.ClusteringArrayPath.replaceName(filterArgs.value<std::string>(k_FeatureIdsArrayName_Key));
   inputValues.FeatureAM = filterArgs.value<DataPath>(k_FeatureAMPath_Key);
   inputValues.ParseOrder = filterArgs.value<ChoicesParameter::ValueType>(k_ParseOrderIndex_Key);
-  inputValues.Seed = filterArgs.value<std::mt19937_64::result_type>(k_SeedValue_Key);
+  inputValues.Seed = seed;
 
   return DBSCAN(dataStructure, messageHandler, shouldCancel, &inputValues)();
 }

src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.hpp

@@ -9,7 +9,7 @@ namespace nx::core
 {
 /**
  * @class DBSCANFilter
- * @brief This filter will ....
+ * @brief Clusters 2D or 3D point data using Grid-based DBSCAN (GDCF), grouping points by spatial density into labeled clusters with noise points assigned cluster ID 0.
  */
 class SIMPLNXCORE_EXPORT DBSCANFilter : public IFilter
 {