KTestify cucumber plugin HTTP client

[williamgarr]

Hello, in my app I have a content chain of consumers/producers. But a part of the chain is triggered with API calls instead of Kafka.

Ideally I want to test it with the rest of the content chain in my KTestify cucumber setup.

The first workaround tooling I thought of was :

• A generic script to launch API calls using curl
• The script is run by the KTestify cucumber run script step

That could work but it is IMO a little dirty.

An elegant solution could be :

• A dedicated HTTP client KTestify plugin
• Which adds specific steps to launch HTTP requests during the tests
• It could improve readability in the tests and also technical integration complexity due to custom scripts
• Also we can imagine more complex use cases in the future involving HTTP requests

Hope you like the idea ;)

[nil-malh]

Hello @williamgarr. Excellent idea, here's the design I thought of, feel free to give me your feedback!

Project structure

Rather than bolting HTTP onto the existing Kafka-shaped RecordFetcher (poll-until-appears), I want ktestify-core to get a second, sibling transport contract for synchronous request/response transports. ktestify-plugin-http then becomes a thin adapter on top of it, same pattern as ktestify-plugin-azureblob today.

ktestify-core (additions)
└── io.github.ktestify
    ├── io/core/
    │   ├── RequestResponseClient<Req, V>        — NEW sibling to RecordFetcher<V>: execute(Req) → List<ConsumedRecord<V>>
    │   ├── AbstractSynchronousConsumer<Req, V>  — NEW orchestration base, mirrors AbstractKafkaConsumer
    │   └── PollingRequestResponseClient<Req, V> — NEW generic retry/poll decorator ("eventually" semantics)
    ├── models/
    │   └── ConsumedRecord<V>                    — + attributes: Map<String,String> (statusCode, elapsedMs, etc.)
    └── match/
        ├── MatchContext                         — + expectedAttributes: Map<String,String>
        ├── RecordMatcherFactory                  — + METHOD_MATCH_ATTRIBUTES
        └── impl/AttributeRecordMatcher<V>        — NEW generic matcher (status code / header / any attribute)

ktestify-plugin-http (new module, scaffolded from ktestify-plugin-archetype)
└── io.github.ktestify.http
    ├── HttpPlugin                       — KtestifyPlugin SPI impl
    ├── config/HttpConfig                — ktestify.plugins.http.*
    ├── entities/KtestifyHttpEndpoint    — registered endpoint (alias, baseUrl, headers, auth)
    ├── io/
    │   ├── HttpRequestSpec              — method/url/headers/queryParams/body
    │   └── HttpRequestResponseClient    — implements RequestResponseClient<HttpRequestSpec, String>
    │                                       (java.net.http.HttpClient, already used in ktestify-plugin-notifications)
    ├── HttpConsumer                     — extends AbstractSynchronousConsumer<HttpRequestSpec, String>
    └── steps/
        ├── HttpBackgroundSteps          — @Given: register endpoint / auth
        ├── HttpActionSteps              — @When: send request, store response under alias
        ├── HttpValidationSteps          — @Then/@And: status / body / header assertions
        └── SharedHttpResources          — PicoContainer DI: ObjectManager<KtestifyHttpEndpoint> + ObjectManager<ConsumedRecord<String>>

Key point: the HTTP response is stored as a plain ConsumedRecord<String> (body = value, response headers = headers, status code = the new attributes map). That means all existing matchers work unchanged: FileRecordMatcher, XmlRecordMatcher, XPathRecordMatcher, FieldsRecordMatcher, for body assertions. Only status/header/metadata checks need the one new generic AttributeRecordMatcher.

Gherkin Syntax

Background

Given HTTP endpoint
  | endpointAlias | baseUrl                    | defaultHeaders                        |
  | orders-api    | http://localhost:8080/api  | Content-Type=application/json         |

Given HTTP bearer token
  | endpointAlias | token          |
  | orders-api    | {{ENV:API_TOKEN}} |

Action

When HTTP request is sent
  | endpointAlias | method | path             | file          | responseAlias  |
  | orders-api    | POST   | /orders/validate | order.json    | validate-resp  |

When HTTP request is sent
  | endpointAlias | method | path         | queryParams        | responseAlias |
  | orders-api    | GET    | /orders/{id} | id=ORD-001         | fetch-resp    |

Validation

Then expected HTTP response status
  | responseAlias | statusCode |
  | validate-resp | 200        |

Then expected HTTP response body from file
  | responseAlias | file            | excludedKeys   |
  | validate-resp | expected.json   | timestamp,id   |

And HTTP response header should match
  | responseAlias | header       | value            |
  | validate-resp | Content-Type | application/json |

# Eventually-consistent polling, mirrors "record should appear in topic"
Then HTTP endpoint should eventually return
  | endpointAlias | method | path         | expectedStatus | readTimeout |
  | orders-api    | GET    | /orders/ORD-001/status | 200  | 30          |

Configuration

ktestify.plugins.http {
  connect-timeout   = 10s
  read-timeout      = 30s   ; overridable per-step via readTimeout column
  poll-interval     = 500ms
  follow-redirects  = true
  default-headers { }
  tls {
    trust-all = false       ; explicit opt-in only, for local/dev endpoints
  }
}

Notes, why this needs core work, not just a plugin

ktestify-core currently only knows one transport shape: RecordFetcher<V>.fetch(), a no-arg call that blocks until a record shows up on a background stream (Kafka topic, Azure Blob polling). That's a perfect fit for "wait for the consumer to eventually produce something", but it's the wrong shape for HTTP: an HTTP call is caller-initiated and synchronous, you send a request right now and get an answer immediately, you don't "poll until a request appears".

My first instinct was to just write the HTTP plugin with its own HttpActionService/HttpValidationService pair and skip RecordFetcher entirely, but that means the plugin stops following the core API and starts reinventing its own parallel action/validation stack, which defeats the point of having a shared architecture (and won't help the next synchronous transport either, gRPC, SOAP, whatever comes up).

So instead I want to add a proper second contract to core, RequestResponseClient<Req, V>, sitting right next to RecordFetcher<V>. It has the same execute(...) → List<ConsumedRecord<V>> return shape, so it plugs into the exact same assertion layer (RecordMatcher, MatchContext, MatchResult, RecordMatcherFactory) with zero new matcher code needed for body comparisons. The only genuinely new piece is a small, transport-agnostic AttributeRecordMatcher for status-code/metadata checks (reusable later by gRPC status codes, MQ reason codes, etc.), plus an AbstractSynchronousConsumer orchestration base that mirrors AbstractKafkaConsumer 1:1.

Net effect: ktestify-plugin-http ends up being a thin adapter, exactly like ktestify-plugin-azureblob is today, instead of a one-off that quietly steps outside the framework's own rules. A bit more upfront work in core, but it pays for itself the moment we want a second synchronous transport.

I've written up the full implementation spec (core changes first, then the plugin). I will link the issues when I finished creating them :).

What do you think ?
Cheers !
Nil

[nil-malh]

Issue for the ktestify-core API change : #55