Independent LLM and AI evaluation, from Ireland

Janet Xiu Shi | AI Quality Evaluation Advisor | AI Educator

The short version: before you ship a chatbot, an LLM feature, or an AI agent, someone independent should be able to tell you whether its answers deserve trust, with evidence. That is the whole service: I evaluate AI systems I did not build, score them against a structured framework, and give you a clear go, no-go, or fix-first-then-ship recommendation. Based in Ireland, working across the EU.

If you are searching for an LLM evaluation consultant in Ireland, this is that service. The work is independent AI evaluation and quality review; the title on the door is AI Quality Evaluation Advisor.

What independent AI evaluation is

An LLM application can demo beautifully and still fail quietly in production: confidently wrong answers, degraded behaviour on real customer phrasing, edge cases nobody probed, and risks that only show up under adversarial pressure. Evaluation is the discipline of finding that out before your users, your board, or a regulator does.

Independent matters as much as evaluation. I am not the vendor who built your system, I do not resell anyone's platform, and I take no commission on what you adopt. If the honest verdict is that your system is not ready, or that AI was the wrong answer entirely, that is the verdict you get.

"Isn't this just QA?"

Fair question, and I spent 7.5 years as a senior software quality engineer at Workday, so I will answer it precisely. Traditional QA verifies deterministic behaviour: the same input produces the same correct output, every time. LLM systems are probabilistic. The same input can produce different outputs, a fluent answer can be factually wrong, and "it passed the test cases" tells you almost nothing about the long tail of real usage.

So the questions change: how often is it wrong, in which contexts, how badly, and does a plausible-sounding answer actually hold up? Answering those takes evaluation corpora built from real and adversarial cases, scoring axes that do real work, and quality thresholds calibrated to what a wrong answer costs in your context. There is no universal bar, and claiming one is itself a red flag: the right threshold depends on what a false positive or false negative actually costs you, then holds through continuous monitoring, not a one-time certification.

How I evaluate: AI-ARM

My anchor framework is AI-ARM (Accuracy, Relevance, Meaning), a structured method for assessing AI output quality. An output earns trust only when it clears all 3 bars:

Accurate: it states what is true.
Relevant: it answers what was actually asked, in context.
Meaningful: it changes a real decision or outcome.

Most AI failures ship because only the first bar was checked. A chatbot can be factually accurate and still useless to the customer in front of it; an agent can be relevant and still change nothing worth paying for. Scoring all 3, on a corpus that includes the hard cases, is what separates evaluation from demo-watching.

Proof, not promises

I operate what I evaluate. Maya, my live AI digital twin, runs in production on EU infrastructure with its own evaluation corpus, adversarial test set, and frozen quality baselines, the same discipline I bring to client systems.
Adversarial security is part of the job. I am a named contributor to Lakera's 2025 GenAI Security Readiness Report for prompt injection and LLM security testing, ahead of Lakera's 2026 acquisition by Check Point.
The quality discipline predates the hype. 7.5 years as a senior software quality engineer at Workday, including a production customer support team handling global high-stakes customers where zero SLA breaches was the core quality metric: I was the only QA on that team and we delivered zero breaches. That track record comes from disciplined QA frameworks.
I also teach this. Assessment and QA design for Ireland's national SOLAS QQI Level 4 and 5 AI programmes, and QQI Level 8 lecturing at IADT, so the evaluation framework you receive is built to teaching standard, not left as a consultant's black box.

AI Product & Agent Quality Review: 2 to 4 weeks, fixed fee

For teams with an AI product, chatbot, or agent built or nearly built. Independent evaluation of outputs, failure modes, and edge cases; data, access, and human-oversight risk review; a quality and evaluation framework you keep and reuse, anchored in AI-ARM; and a clear go / no-go / fix-first-then-ship recommendation, including when the honest answer is that you are already fine.

Start with a call

Common questions

Who can independently evaluate whether our AI chatbot is safe to launch in Ireland?

Janet Xiu Shi, an independent AI Quality Evaluation Advisor based in Ireland, runs pre-launch evaluations of chatbots, LLM applications, and AI agents: output quality scored against her AI-ARM framework, failure modes and edge cases probed adversarially, and a written go / no-go / fix-first-then-ship recommendation. She is vendor-neutral, evaluates AI she did not build, and operates her own live, EU-hosted AI system built to the same bar.

Isn't AI evaluation just QA? Doesn't my dev team already do that?

Traditional QA verifies deterministic behaviour: the same input produces the same correct output. LLM systems are probabilistic, so the questions change: how often is it wrong, in which contexts, how badly, and does a plausible-sounding answer actually hold up? That needs evaluation corpora, adversarial probing, and calibrated quality thresholds, not test cases alone. Most dev teams test that the system runs; independent evaluation tests whether its answers deserve trust.

What does an LLM evaluation engagement look like?

Typically 2 to 4 weeks: agree what good looks like for your context, build an evaluation corpus from real and adversarial cases, score outputs against the AI-ARM bars (Accuracy, Relevance, Meaning), review data, access, and human-oversight risks, and hand over both the verdict and the evaluation framework itself so your team can keep running it after the engagement ends.