Scoring
Score a bug report’s usefulness
Combine specific quality judgments with weights you control.
01 The questions
How usable are this report’s reproduction steps?
- 0
- No steps or triggering context.
- 1
- Some context, but important steps are missing.
- 2
- Clear steps and environment sufficient for an initial attempt.
How clearly does the report distinguish observed and expected behavior?
- 0
- Neither is clear.
- 1
- One is clear, but the other is missing or vague.
- 2
- Both are specific and distinguishable.
Copy includes the complete instructions, criteria, usage notes, and attribution.
02 Input
A bug report, including reproduction steps, environment, and observed behavior.
{
"report": "On Android 15, open Settings and tap Export. The app closes before showing the share sheet. Expected: a CSV export. Reproduced three times on version 2.4."
}03 Answer & policy
Two expected scores on a 0–2 rubric, each with confidence. Scores can be fractional.
Your code decides what happens next.
The example normalizes each score to 0–1 and weights reproducibility 60%, behavior 40%. Low-confidence judgments go to review before ranking.
04 Use it in your code
Node.js 24 · TypeSafe SDK 0.6.0 · Set TYPESAFE_API_KEY in your environment. Run on your server; API calls incur provider charges.
import { score, TypeSafeClient, type JsonValue } from "@typesafe-ai/sdk";
// Illustrative input, not a recorded model test.
const state: JsonValue = {
"report": "On Android 15, open Settings and tap Export. The app closes before showing the share sheet. Expected: a CSV export. Reproduced three times on version 2.4."
};
const client = new TypeSafeClient();
try {
const response = await client.systemOne({
model: "jev-latest",
state,
questions: {
reproduction: score(
"How usable are this report’s reproduction steps?",
[
"No steps or triggering context.",
"Some context, but important steps are missing.",
"Clear steps and environment sufficient for an initial attempt."
]
),
behavior: score(
"How clearly does the report distinguish observed and expected behavior?",
[
"Neither is clear.",
"One is clear, but the other is missing or vague.",
"Both are specific and distinguishable."
]
)
},
});
const { reproduction, behavior } = response.answers;
const review = Math.min(reproduction.confidence, behavior.confidence) < 0.7;
const usefulness = 0.6 * (reproduction.score / 2) + 0.4 * (behavior.score / 2);
console.log({ usefulness, action: review ? "review_scores" : "rank_report" });
} catch (error) {
console.error("Decision unavailable; use your fallback or human review.", error);
process.exitCode = 1;
}The wrapper and example input are provided by Jev Directory. Checked against SDK types; no live model call was made. Pin a model version before evaluating production behavior.
Before you adapt it
- Weights are editorial choices, not model measurements.
- A well-written report can still describe an incorrect or low-impact issue.
More about the original project or pattern
What it does
Composite scoring breaks a broad evaluation into separate dimensions, asks a narrow Score question for each one, and combines the resulting values with a formula in application code. This avoids hiding business priorities inside one compound question.
What you can reuse
Reuse the separation between judgment and weighting. Jev evaluates well-defined rubrics; code normalizes and combines the outputs. When priorities change, developers can alter a coefficient or decision boundary without rewriting every question or rerunning unrelated dimensions.
How it fits
One shared state is evaluated against several independent Score questions. Their typed results enter a deterministic formula, followed by sorting, routing, or a policy threshold. The formula remains inspectable and testable without a model call.
Setup and compatibility
The pattern is documented for the TypeSafe SDK. Current syntax and model selection should come from the official quick start and the specific composite-scoring page.
Limitations
Weights express product policy; they do not correct a weak rubric or unrepresentative evaluation data. Normalization must respect each scale, and a weighted total can hide a disqualifying condition. Use an explicit veto question when one dimension must never be averaged away.
Sources
Primary source: TypeSafe composite scoring.