SED-4885 Creating plan samples for RPA by jeromecomte · Pull Request #32 · exense/step-samples

jeromecomte · 2026-08-21T14:09:36Z

No description provided.

…before

…bout throughput measurements

… sample

david-stephan

Overall good, but took me quite some time to go through it, added several comments. Note, I skipped all keywords and stopped checking the YAML at some points.

david-stephan · 2026-08-24T06:54:27Z

+| `Demo_Google-search.json` | A sequence calling an Echo keyword and asserting on its output |
+| `Demo_Data-driven.json` | A data-driven plan iterating over a data source |
+
+Plans built in the Step plan editor are stored serialized as JSON, and these two files are


I'm not sure how the plan are stored in DB are relevant, I would just say that this is the format used for import/export and not meant for authoring

Totally agree. Will be changed

david-stephan · 2026-08-24T07:06:43Z

@@ -0,0 +1,67 @@
+# A standalone YAML plan: the same tree you would write inside an automation


The wording is not very clear for me. Not perfect either, but another propostion:

==============================================================================

STANDALONE PLAN

This file represents a single, self-contained Step plan.

Usage:

• Upload directly: Import into the Step UI via "Add plan" > "Create from YAML".

• Convert/add to a package: Move the contents under a plans: list entry inside

an automation package manifest.

This file is a syntax illustration, not a runnable plan - the keywords it

calls do not exist. For plans you can execute, see ../rpa/

==============================================================================

Agree with your suggestion. Will be changed

david-stephan · 2026-08-24T07:17:25Z

+          keyword: "Submit Record"
+          inputs:
+            - literalInput: "abc"              # static
+            - numericInput: 777                # static number


This is still a string not sure the comment is correct/clear:

To be discussed

david-stephan · 2026-08-24T07:38:17Z

+
+## What these samples teach
+
+The controls are the vocabulary; these are the ideas that decide whether a plan is any good.


I would write "The controls provide the vocabulary; the following principles determine whether a plan is well-designed:"

david-stephan · 2026-08-24T07:42:13Z

+   is the reason to use a plan at all instead of one large script.
+
+6. **Values come from outside the plan.** The same bot serves a person on demand
+   (execution parameters) and a nightly `schedules` entry, with credentials held in


not sure it's clear what we means with (execution parameters) a concrete example would help; also not sure that the distinction between on-demand run and scehduled ones belongs to *Values come from outside the plan.

david-stephan · 2026-08-24T08:46:21Z

+It governs whether one failing cleanup step stops the **remaining** cleanup steps — the same
+"keep going" semantics as on a sequence, applied to the cleanup list. Without it, a failing


"same "keep going" semantics as on a sequence", I have the feeling we did not mention continueOnError for sequence before, so that's a bit confusing

david-stephan · 2026-08-24T08:53:14Z

+session.
+
+Put a `sequence` inside the session and use **its** `before` / `after` instead. Those steps
+run within the session's token, and `after` still runs when the body fails — so cleanup is


"token" is probably not required, it just runs outside of the session. I guess all this is (and some other sections of this PR) comes from claude trying something, figuring it out it's not working as expecting and documenting it as such. While this is good as other user will fall in the same traps, I feel the wording around this a bit unnatural.

This is exactly the case and I had to perform several iterations to remove such "logs". To be fixed

david-stephan · 2026-08-24T08:57:08Z

+| Mechanism | Returns outputs? | Use it when |
+|-----------|------------------|-------------|
+| **Composite keyword** | Yes, via `return` | The thing you are reusing is a *step* inside someone else's plan. **Reach for this first** |
+| `callPlan` | No — it produces its own report branch | The thing you are reusing is a whole bot in its own right, with its own schedule |


I don't get " own right, with its own schedule"

Don't get it neither ;) To be fixed

david-stephan · 2026-08-24T09:00:12Z

+
+## `forEach threads` vs `testSet threads`
+
+- `threads` on a **loop** parallelises **rows** of one bot.


would say "row processing" or something

david-stephan · 2026-08-24T09:02:47Z

+
+## A plan that is called is still a plan
+
+`Shared - Archive processed records` exists to be called by plan B, but nothing marks it as


this might also be confusing for readers. Are seems to be more an artefact of "this is a sample, it should not fail". In real life, if I define such a shared plan, nothing say it has to run standalone too and ahve default values.

Agree. We should clearly mark this as an requirement for this sample or mark this plan as expected to fail

… content

david-stephan · 2026-09-04T12:39:06Z

 |--------|-----------|----------|----------|-------|
 | [rpa-selenium](rpa-selenium/) | selenium | java | keyword-driven | intermediate |

+> The sample above is a complete RPA project. For focused examples of the **plan** itself —


" of the plan itself" sounds weird, especially singular in this context.

david-stephan · 2026-09-04T12:58:33Z

+        children:
+
+          # -------------------------------------------------------------
+          # THE ITERATION IS THE UNIT THE LOAD NUMBERS ARE COUNTED IN.


I don't understand this sentence and honestly not much of the paragraph until the line 65. This sample also only introduce number of users and iterations stated the more complex "knobs" will be in the 2nd sample, but ere we already talk about pacing and throughput.

david-stephan · 2026-09-04T13:03:39Z

+                      # assert the response times would look excellent
+                      # and mean nothing.
+                      #
+                      # This runs once per iteration, so a failure at


I find speaking about iteration here is confusing it runs for every Checkout keyword execution. I would also add that all keyword response should be asserted, not just the final one in the workflow.

david-stephan · 2026-09-04T13:06:58Z

+        # THE SLA, EXPRESSED IN THE PLAN.
+        #
+        # `performanceAssert` compares an aggregate of a measurement against a
+        # threshold once the load is over. It turns a load test from "here are


once the "test" is over ?

david-stephan · 2026-09-04T13:09:23Z

+        # TWO RULES ABOUT WHERE IT GOES, both easy to get wrong:
+        #
+        #  1. A `performanceAssert` MUST live in an `after` or `afterThread`
+        #     block. As an ordinary child - of the thread group, of a testCase,


Writing down the error you will get if you miss place it feels quite redundant.

david-stephan · 2026-09-04T13:23:53Z

+          continueOnError: true
+          steps:
+            # "Purchase journey" is created by the keyword with
+            # startMeasure/stopMeasure, so - unlike an instrumented sequence -


not sure it's worth repeating the limitation here

david-stephan · 2026-09-04T13:25:30Z

+                expectedValue: 250
+
+  # =========================================================================
+  # C. Measuring the failures, not just the successes.


What's the added value compared to the sample 01 ?

david-stephan · 2026-09-04T13:26:21Z

+   virtual user.
+2. **`measurementName` must name a keyword or custom measurement**, never an `instrumentNode` one —
+   that fails with `No measurement is matching the defined filters.`, the same message a misspelled
+   name gives. For an SLA on a multi-step transaction, emit a custom measurement — see


I only checked the sample 01 and 05 at the time of writing this, but composite keyword could be used instead

david-stephan · 2026-09-04T13:34:37Z

+| Instrumented node | the node's `nodeName` | `instrumentNode: true` | **no** — dashboards only |
+| Custom | whatever the keyword chooses | `output.startMeasure(...)` in the keyword | **yes** |
+
+Custom measurements are the only way to time something **smaller** than a keyword call (two page


this paragraph is quite hard to read/understand, custom keyword measurement is to measure part of the keyword execution...
Also written somewhere else "For an SLA on a multi-step transaction, wrap it in a custom measurement" sound wrong, You basically have to create a keyword containing the multi steps first. If it contains only these multisteps you're done, if it contains more than you need a custom measurement to only measure what you need.

david-stephan · 2026-09-04T13:56:18Z

+
+### Measure the failures, not just the successes
+
+An average over successful calls only is the most misleading number in load testing — when a system


This sentence is confusing and I saw the same pattern in the automation package YAML (probably in sample 01 too).
What would be incorrect is to have 1. no functional assertion and 2. rely only on response times without considering failure and success rate.
I see that Performance assert doesn't support rate and doesn't support check on the status at all, but what is formulated here sounds even more confusing than being transparent about it. Using the COUNT in performance assert given that it includes all statuses don't help to automatically assert the test results.

SED-4885 Creating plan samples for RPA

8fe5261

jeromecomte requested a review from david-stephan August 21, 2026 14:19

jeromecomte added 6 commits August 21, 2026 22:51

SED-4885 Creating plan samples for load testing (draft)

0eb0430

SED-4885 Creating plan samples for load testing - Precising iterations

1a9dbc1

SED-4885 Creating plan samples for load testing - Moving dataSets to …

29687c7

…before

SED-4885 Creating plan samples for load testing - Fixing register slips

829c5a1

SED-4885 Creating plan samples for load testing - Improving clarity a…

cd3b63c

…bout throughput measurements

SED-4885 Creating plan samples for load testing - Remove assertMetric…

74a2250

… sample

david-stephan requested changes Aug 24, 2026

View reviewed changes

jeromecomte added 2 commits August 24, 2026 11:19

SED-4885 Creating plan samples for load testing - Removing duplicated…

3c50835

… content

SED-4885 PR Feedbacks RPA Samples

b89c699

david-stephan requested changes Sep 4, 2026

View reviewed changes

		@@ -0,0 +1,67 @@
		# A standalone YAML plan: the same tree you would write inside an automation


		## What these samples teach

		The controls are the vocabulary; these are the ideas that decide whether a plan is any good.

		It governs whether one failing cleanup step stops the remaining cleanup steps — the same
		"keep going" semantics as on a sequence, applied to the cleanup list. Without it, a failing


		## `forEach threads` vs `testSet threads`

		- `threads` on a loop parallelises rows of one bot.


		## A plan that is called is still a plan

		`Shared - Archive processed records` exists to be called by plan B, but nothing marks it as


		### Measure the failures, not just the successes

		An average over successful calls only is the most misleading number in load testing — when a system

Uh oh!

Conversation

jeromecomte commented Aug 21, 2026

Uh oh!

david-stephan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeromecomte Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

==============================================================================

STANDALONE PLAN

This file represents a single, self-contained Step plan.

Usage:

• Upload directly: Import into the Step UI via "Add plan" > "Create from YAML".

• Convert/add to a package: Move the contents under a plans: list entry inside

an automation package manifest.

This file is a syntax illustration, not a runnable plan - the keywords it

calls do not exist. For plans you can execute, see ../rpa/

==============================================================================

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jeromecomte Aug 24, 2026 •

edited

Loading

• Convert/add to a package: Move the contents under a `plans:` list entry inside