Skip to content

Latest commit

 

History

14 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HO1 — LLM Evaluation Report

Hands-on portfolio project · Week 1 · Solo · module M4. Part of the ForgeMind AI — AI Productivity Essentials course.

Goal

Done when: you have run one real task through two or more local models and filled in a report card scoring each on tokens, time and accuracy — then picked the best local model, backed by your own numbers.

What to ship

Your completed report card (CSV / Excel): one row per test item per model, with totals and averages, and the model you chose.

There is no code to write and nothing to deploy. The report card is the deliverable.

What you measure

Run the same task through two or more local models in LM Studio — entirely offline. No API key, no cloud, no Claude in the loop.

Metric Where it comes from
Accuracy You score each item yourself — right or wrong
Tokens Shown by LM Studio under each reply
Time How long the reply took

Pick a problem statement

Each one is a ready-to-run task with 15 test items. These five are starting points — feel free to use any other problem statement of your own (a text task on your own data works great).

  1. Support Tickets — ~150 tickets a week; auto-tag each ticket's category and urgency, offline.
  2. Order Emails — orders arrive as plain-text emails; pull out the customer name, item and quantity.
  3. Meeting Notes — you drown in call recordings; turn transcript snippets into an action-item list with owners.
  4. Social Comments — a flood of comments and reviews; tag each by sentiment so you can reply fast.
  5. Email Drafting — repetitive customer emails; have a local model draft the first reply.

How it works

  1. Pick a task and open its sample — each lists 15 items to test. Or use your own problem statement.
  2. In LM Studio, load two or more local models and run all 15 through each — a fresh chat per item, so earlier items don't fill the context and waste tokens.
  3. As you go, note each item's tokens, time and score (right or wrong) in the report card.
  4. Download the report card (CSV), fill it in, and submit it — that's your deliverable.

About this repo

This repo is a reference, not a template. The five samples and the report card live here so you can open them and run the task. You do not need to fork it or push anything — your deliverable is the filled-in report card.


HO1 · Solo · ForgeMind AI Course · module M4 (Week 1)

🖥️ This hands-on is 100% offline

Everything here runs on your own machine with LM Studio — no API key, no cloud, no Claude in the loop.

About

HO1 — LLM Evaluation Report · ForgeMind AI hands-on (Week 1)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages