Skip to content

Create Developer Documentation for Evaluation Features #796

Description

@maxtechera

Objective

Create developer documentation for extending and integrating with evaluation features.

Tasks

API Documentation

  • Document all dataset API endpoints with examples
  • Document all evaluator API endpoints with examples
  • Document evaluation execution endpoints
  • Document request/response schemas
  • Add curl examples for common operations

Architecture Documentation

  • Document entity relationships (ERD)
  • Document service layer architecture
  • Document evaluation execution flow
  • Document how simple evaluators work
  • Document how LLM evaluators work

Extension Guide

  • Document how to add custom simple evaluators
  • Document how to add custom LLM evaluator templates
  • Document integration points for external evaluation systems
  • Document how to extend evaluation metrics

Integration Examples

  • Show programmatic dataset creation
  • Show running evaluations via API
  • Show retrieving results programmatically
  • Show webhook integration (if supported)

Location

  • packages/server/CLAUDE.md - Add evaluations section
  • packages/components/CLAUDE.md - Add evaluator components section (if applicable)

Acceptance Criteria

  • All API endpoints documented with examples
  • Architecture clearly explained with diagrams
  • Extension guide enables adding custom evaluators
  • Code examples are tested and working
  • Documentation follows existing CLAUDE.md patterns

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions