Skip to content

[UPDATE] Replace output's name with id for data management - #12

Open
kavehshahedi wants to merge 1 commit into
eclipse-tmll:mainfrom
kavehshahedi:replace-name-with-id
Open

[UPDATE] Replace output's name with id for data management#12
kavehshahedi wants to merge 1 commit into
eclipse-tmll:mainfrom
kavehshahedi:replace-name-with-id

Conversation

@kavehshahedi

Copy link
Copy Markdown
Contributor

What it does

Until now, we have utilized output names (e.g., "CPU Usage" or "Disk I/O View") to manage data frames inside the modules. Although this methodology seems to work fine, it becomes problematic for cases where we have multiple outputs with the same name. For instance, when fetching an LTTng trace (kernel + UST) from the trace server, we might get two "Memory Usage" outputs, one is for the kernel's memory usage and the other one comes from the UST.
As of now, we use the output's id for this matter, enabling us to accurately differentiate the outputs from each other as each output has its own unique id.

How to test

You can either run the tests using pytest -v, or check the results/graphs/interpretations of each ML module (e.g., Anomaly Detection, Capacity Planning, etc.).

Follow-ups

N/A

Review checklist

  • As an author, I have thoroughly tested my changes and carefully followed the instructions in this template

@kavehshahedi
kavehshahedi requested a review from bhufmann March 6, 2025 16:46
As now, we use output's id instead of its name for a more robust and
accurate data managements. Hence, for cases with multiple outputs with
same names, the ids will now diffrenciate them.

Signed-off-by: Kaveh Shahedi <kavehshahedi@gmail.com>
@kavehshahedi
kavehshahedi requested a lite review from Copilot and removed request for bhufmann August 27, 2026 23:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the ML modules’ internal dataframe management to key and reference series by output ID instead of output name, addressing ambiguity when multiple outputs share the same name (e.g., multi-trace “Memory Usage” outputs).

Changes:

  • Switched dataframe keys/column references across multiple modules from output name → output ID.
  • Added Experiment.get_output_by_id/get_output_by_name/get_outputs_by_name helpers and introduced duplicate-name disambiguation in assign_outputs().
  • Updated plotting/interpretation code to map IDs back to human-readable names for display.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tmll/ml/modules/root_cause/correlation_module.py Correlation/lag analysis now indexes series by output ID and maps IDs back to names for plot labels.
tmll/ml/modules/resource_optimization/idle_resource_detection_module.py Uses output IDs for stored dataframes and resolves IDs back to names when reporting/plotting.
tmll/ml/modules/predictive_maintenance/capacity_planning_module.py Capacity planning results/plots now use output IDs as keys and map to names for display.
tmll/ml/modules/performance_trend/change_point_module.py Resolves requested metrics to output IDs and maps IDs back to names in plots.
tmll/ml/modules/common/data_fetch.py Renames fetched XY dataframe columns to output ID instead of output name.
tmll/ml/modules/base_module.py Stores processed dataframes under output ID instead of output name.
tmll/ml/modules/anomaly_detection/memory_leak_detection_module.py Refactors Events Table / Memory Usage access to use output IDs (and disambiguate Memory Usage).
tmll/ml/modules/anomaly_detection/anomaly_detection_module.py Plotting now resolves dataframe keys (IDs) back to output names for labels.
tmll/common/models/experiment.py Adds output lookup helpers and disambiguates duplicate output names during assignment.
Suppressed comments (1)

tmll/ml/modules/anomaly_detection/memory_leak_detection_module.py:199

  • analyze_memory_leaks() now only checks for None from _get_events_dataframe()/_get_memory_usage_dataframe(), but those helpers can return an empty DataFrame. In that case _analyze_memory_trend() returns {} and memory_trend["p_value"] will raise KeyError. This should treat empty DataFrames the same as missing data and return a NONE severity result.
        et_df = self._get_events_dataframe()
        if et_df is None:
            self.logger.warning("Events table output not found in the experiment")
            return LeakAnalysisResult(
                severity=MemoryLeakSeverity.NONE,

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 100 to +104
def _post_process(self, **kwargs) -> None:
if "Events Table" in self.dataframes:
df = self.dataframes["Events Table"]
if not all(col in df.columns for col in ["size", "ptr"]):
events_df = self._get_events_dataframe()
if events_df is not None:
events_output = self.experiment.get_output_by_name("Events Table")
id = events_output.id if events_output is not None else None
Comment on lines +133 to +136
if partial_match:
return next((o for o in self.outputs if output_name.lower() in o.name.lower()), None)

return next((o for o in self.outputs if o.name.lower() == output_name.lower()), None)
Comment on lines +270 to +274
if metrics:
for idx, metric in enumerate(metrics):
output = self.experiment.get_output_by_name(metric)
if output is not None:
metrics[idx] = output.id
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants