Scenario-based text systems
Clinical assistants, RAG workflows, documentation models, and coding/CDI models can be assessed when responses can be imported from approved files or collected through an approved OpenAI-compatible chat endpoint.
Technical brief
How a bounded Synset workflow runs.
Synset combines versioned model access, controlled synthetic clinical scenarios, artifact checks, a scoped intervention, and held-out evaluation. Support depends on the model, access path, task, and available evidence.
Current scope
Assessment is broader than hardening.
A model can sometimes be assessed through its outputs even when Synset cannot change it. A hardening pilot requires a reproducible training or adaptation path, a defined output contract, and held-out real evaluation data.
Structured prediction models
Risk models are scoped case by case and require reproducible inference, a defined target, and suitable evaluation data. Longitudinal or survival models require separate review of target, horizon, censoring, and validation design.
API-only or closed systems
A system may be assessed when responses can be captured through an approved path. Hardening is not available without a reproducible customer-approved training or adaptation route.
Partner-driven modalities
Imaging, waveform, voice, and other unsupported modalities are not current general-support claims. They require a separate partner scope and evidence plan.
Workflow
From model contract to retained evidence.
1. Define the contract
Record the model task, declared use, model-facing inputs, expected output, model and data versions, and the evaluation boundary.
2. Build the scenario family
Create controlled synthetic records, timelines, notes, dialogue, or task-specific prompts around a measured weakness. Not every engagement uses every artifact type.
3. Check and freeze artifacts
Apply configured clinical, temporal, support, metadata, and text-grounding checks. Reject failed candidates and version the retained cohort, prompt export, and manifest.
4. Run a bounded intervention
The customer may train or adapt the model. Synset may support a scoped workflow where agreed. Calibration, thresholding, abstention, workflow change, or no intervention may be the correct result.
5. Evaluate and preserve
Compare frozen original and candidate versions on customer-controlled held-out real data, then preserve observed failures as fixed regression cases.
The current controlled probe pilot uses deterministic scenario families when a compatible world-conditioned targeting path is unavailable. Universal autonomous retraining is not a current capability.
Reusable software layer
- versioned probe-suite construction
- prompt export and response import
- OpenAI-compatible response collection where approved
- synthetic scenario generation and artifact checks
- stable manifests and artifact hashes
- response scoring and report generation
- fixed regression-case packaging
Human and partner responsibilities
- intended-use and success-criteria definition
- clinical review and interpretation
- intervention selection
- model-specific integration and training ownership
- regulatory and quality interpretation
- release and deployment decisions
Evaluation discipline
Development and final proof stay separate.
Held-out real data is a customer-controlled evaluation set not used to choose, tune, or select the intervention. Original and candidate model versions are frozen before final evaluation against prespecified success and no-regression criteria.
Subgroup and calibration checks are included where the task and data support them. A non-improving result can lead to more real evidence, a different model, a narrower intended use, or a stop decision.
Evidence record
- declared model use and evaluation scope
- model, data, scenario, and protocol versions
- failure map and what may be addressed
- synthetic-artifact check summary
- frozen intervention record
- held-out comparison when applicable
- remaining risks and regression tests
- claim and use boundaries
Current limitations
Capability remains bounded by access and evidence.
Closed-loop customer-model hardening has not yet been published as a held-out public proof. Current hardening work is a selected design-partner pilot, not a universal product claim.
Institution-local evaluation is an enterprise design-partner path, not a generally available deployment. Synset does not determine clinical deployment or regulatory authorization.
Scope one model and one evaluation question.
Do not send PHI, patient files, model credentials, private checkpoints, or held-out evaluation rows through public email.
Start with readiness