Create an eval case

Adds one case to the suite. Both title and a non-empty steps array are required — steps is optional on the shared case shape so PATCH can be partial, but a case persisted without executable steps could never run. When models is omitted the suite’s configured model is used.

POST /projects/{projectId}/eval-suites/{suiteId}/cases

Create an eval case

cURL

curl --request POST \
  --url https://app.mcpjam.com/api/v1/projects/{projectId}/eval-suites/{suiteId}/cases \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "title": "lists files",
  "steps": [
    {
      "id": "s1",
      "kind": "prompt",
      "prompt": "List the files in the project root."
    },
    {
      "id": "s2",
      "kind": "assert",
      "assertion": {
        "type": "toolCalledWith",
        "toolName": "list_files",
        "args": {
          "args": {}
        }
      }
    }
  ]
}
'

201

Example

{ "id": "<string>", "title": "<string>", "steps": [ { "id": "<string>", "kind": "prompt", "prompt": "<string>", "serverName": "<string>", "toolName": "<string>", "arguments": {}, "action": {}, "assertion": {} } ], "iterations": 5, "isNegative": true, "models": [ { "model": "<string>", "provider": "<string>" } ], "declaredId": "<string>", "expectedOutput": "<string>", "repetitions": 2, "passThreshold": 0.5, "scenario": "<string>", "intent": "<string>", "matchOptions": {}, "checks": { "mode": "inherit", "list": [ {} ] }, "import": { "status": "exact", "sourceCaseKey": "<string>", "note": "<string>" }, "createdAt": 123, "updatedAt": 123 }

Authorizations

Authorization · string · header · required

MCPJam API key (sk_…). Create one at Settings → API keys. Guest sessions cannot use the API, and API keys cannot manage other API keys.

Path Parameters

projectId · string · required

ID of the hosted project that contains the server.

suiteId · string · required

Eval suite ID, as returned by POST /eval-runs.

Body

application/json

Create one case. title and a non-empty steps array are both required — a case persisted without executable steps could never run.

title · string · required

Minimum string length: 1

steps · object[] · required

REPLACES the case's test definition wholesale when provided.

Minimum array length: 1

Show child attributes

id · string

Stable declared id for the case — the id it answers to in a suite file, an import or a CLI argument. Minted for you when omitted, and unique within the suite. Distinct from the id on the response, which is the case's platform row id.

Pattern: ^[A-Za-z0-9_-]{1,128}$

expectedOutput · string

iterations · integer

Required range: 1 <= x <= 10

repetitions · integer

Per-case repetitions under verdict policy 2; omitted cases inherit the suite default.

Required range: 1 <= x <= 100

passThreshold · number

Per-case pass threshold fraction under verdict policy 2.

Required range: 0 <= x <= 1

isNegative · boolean

scenario · string

intent · string

Optional authored analytics grouping label. Must be already trimmed.

Required string length: 1 - 64

Pattern: ^\S(?:[\s\S]*\S)?$

models · object[]

Show child attributes

matchOptions · object | null

null clears the case override and inherits the suite's.

checks · object | null

null clears the case override.

Show child attributes

import · object

What a converter CLAIMED about one imported case. exact is CONVERTER-CLAIMED exact — the converter says it applied a structural mapping rule, cited in note. MCPJam has NOT verified semantic equivalence, so user-facing copy must say "claimed exact", never "verified" or "accepted". Claim-only: who approved an approximation, when, and why is a PER-RUN decision frozen on the run (ImportEligibility.approvedApproximationReceipts), never stored on the case — an approval that lived on a case would outlive the run it was granted for and the edit that invalidated it. Approval and internal keys are rejected with 400, never stripped.

Show child attributes

Response

201

application/json

The created case.

A persisted eval case, in the public steps-first shape. Note this is NOT EvalTestCase, which is the INLINE authoring shape accepted by suite creation.

id · string · required

title · string · required

steps · object[] · required

Ordered test steps. A prompt step is a model turn; a single model-free toolCall step is a render-check; assert steps hold the expectations.

Minimum array length: 1

Show child attributes

iterations · integer · required

Required range: 1 <= x <= 10

isNegative · boolean · required

When true, the case passes if NO tools are called.

models · object[] · required

Show child attributes

declaredId · string

The case's effective declared id. Absent on cases authored before declared identity existed.

expectedOutput · string

repetitions · integer

Trials this case runs under verdict policy 2, overriding the suite default. Absent means the case inherits it. NOT a second spelling of iterations: that one is the legacy count, which the legacy resolver reads as a FLOOR (max(iterations, suite.minimumIterations)) and which a policy-2 case still reports for compatibility. This one is exact.

Required range: x >= 1

passThreshold · number

Fraction of this case's trials that must pass, overriding the suite default. Absent means the case inherits it. Never derived from the suite's minimumAccuracy, which is a PERCENT under a different resolver.

Required range: 0 <= x <= 1

scenario · string

intent · string

Optional authored analytics grouping label. Must be already trimmed; absent means unlabelled.

Required string length: 1 - 64

Pattern: ^\S(?:[\s\S]*\S)?$

matchOptions · object

Absent when the case sets none — omitted from the response rather than sent as null.

checks · object

Absent when the case sets none — omitted from the response rather than sent as null.

Show child attributes

import · object

What a converter CLAIMED about one imported case. exact is CONVERTER-CLAIMED exact — the converter says it applied a structural mapping rule, cited in note. MCPJam has NOT verified semantic equivalence, so user-facing copy must say "claimed exact", never "verified" or "accepted". Claim-only: who approved an approximation, when, and why is a PER-RUN decision frozen on the run (ImportEligibility.approvedApproximationReceipts), never stored on the case — an approval that lived on a case would outlive the run it was granted for and the edit that invalidated it. Approval and internal keys are rejected with 400, never stripped.

Show child attributes

createdAt · number | null

updatedAt · number | null