AI Security & Prompt Evaluation Lab
Independent Project / AI Security / Cybersecurity / Responsible AI
A defensive AI assurance portfolio lab for evaluating assistant behavior before release using synthetic probes, transparent deterministic scoring, human-review workflows, threat modeling, audit evidence, and responsible AI governance.
Problem or objective
Teams deploying AI assistants need repeatable, explainable pre-release assurance across instruction robustness, data protection, policy consistency, and tool safety. Without a shared evaluation method, results are anecdotal and cannot be reviewed or repeated.
Rachel's role
Sole designer and full-stack builder responsible for the evaluation model, synthetic probe catalogue, deterministic scoring logic, dashboards, human-review workflow, threat model and governance documentation, testing, and public documentation.
Process
- 01Assurance requirements and recruiter-facing positioning
- 02Evaluation domain definition across instruction robustness, data protection, policy consistency, and tool safety
- 03Typed synthetic fixture and probe catalogue design
- 04Deterministic 0-100 scoring and risk-band logic
- 05Run history and domain dashboard design
- 06Human-review queue and verdict routing
- 07Threat modeling and trust-boundary documentation
- 08Audit evidence and limitation disclosures
- 09Testing, linting, and production build verification
Architecture components
Fixtures and probes
- Typed synthetic fixtures
- Synthetic probe catalogue
- Domain tagging
Evaluation engine
- Deterministic scoring logic
- Risk bands
- Conservative verdict logic
Review and accountability
- Human-review queue
- Review routing indicators
- Audit evidence records
Reporting
- Domain dashboards
- Run history
- Limitation and boundary documentation
Case study
Business problem
Teams deploying AI assistants need repeatable, explainable pre-release assurance across instruction robustness, data protection, policy consistency, and tool safety. This lab turns those concerns into a single evaluation workflow that can be repeated and reviewed.
Architecture approach
Typed synthetic fixtures feed deterministic scoring logic, which drives domain dashboards, a human-review queue, and audit evidence. Trust boundaries are documented so it is clear what the evaluation observes and what it never touches.
Evaluation methodology
- Synthetic probe catalogue organized by evaluation domain
- Run history capturing each evaluation pass
- Transparent 0-100 scoring with the inputs visible
- Risk bands derived from scores rather than opinion
- Conservative verdict logic that fails toward review
- Review routing for anything below confident thresholds
Security and responsible AI
- Threat model covering instruction robustness, data protection, policy consistency, and tool safety
- Data-protection boundaries: no live model APIs and no real user data
- Human accountability required before any verdict is treated as final
- Explicit limitations: the demo is not a security certification and says nothing about any real AI product
Engineering quality
- TypeScript, React, and TanStack Start with typed domain modules
- Vitest suite green at 27 of 27 passing
- Reusable dashboard components across evaluation domains
- Lint clean with 0 errors and only pre-existing shadcn Fast Refresh warnings
- Successful production build
Implemented versus production boundary
- Implemented: synthetic fixtures, deterministic scoring, review indicators, domain dashboards, documentation, and tests
- Production would require real model and provider integrations
- Production would require authentication, protected secrets, and production authorization
- Production would require persistent storage and monitored evaluation infrastructure
- Production would require operational governance workflows with named owners
Tools and technologies
- TypeScript
- React
- TanStack Start
- Tailwind CSS
- Vitest
Security and ethical considerations
- Synthetic evaluations only; no live model APIs and no real user data
- No operational instructions for bypassing safety controls; the lab is defensive assurance only
- Documented trust boundaries and data-protection limits for every evaluation path
- Human accountability required for verdicts; scoring is advisory and deterministic, never autonomous enforcement
- Explicit implemented-versus-production boundaries on every control
- No secrets, provider keys, or environment credentials in the public repository
Outcome
A public, documented AI assurance lab with domain dashboards, review workflows, and governance documentation, verified at 27 of 27 automated tests passing, lint clean with 0 errors, and a successful production build.
Portfolio lab using synthetic evaluations only. No live model APIs, no real user data, and no operational instructions for bypassing safety controls. This lab is not a security certification and is not evidence about the safety of any real AI product.