Machine Learning Fine-Tuning & Model Evaluation

What it does

Machine Learning Fine-Tuning & Model Evaluation is a Cursor skill: a SKILL.md playbook that AgenticKit installs at .cursor/skills/ml-training. Cursor's agent loads it when the task matches the skill description, instead of stuffing the same rules into every chat.

Practices for preparing JSONL datasets, fine-tuning open-source LLMs (Llama/Mistral) using LoRA, and benchmarking output quality.

LoRA fine-tuning, dataset cleaning, loss curves, and evaluation benchmarks. The file is domain knowledge, not a slash macro. You do not type it. The agent reads it when the job is ml training.

Why it exists

Default Cursor has no memory of how your team ships code, schema, and tests. Founders paste the same conventions into chat, then watch the model drift on the next turn. This skill exists so those conventions live on disk and load only when relevant.

It ships in the Engineering Kit. It is wired to the Machine Learning Engineer (ml-engineer) and AI & Vector Engineer (ai-engineer) agents. It is wired to the /ml-finetune: ML Model Fine-Tuning & Dataset Pipeline and /ai-feature: AI Integration & Vector Pipeline Builder commands. That graph is the point: the skill is the standard, the agent is the role, the command is the trigger.

When to use it

• Use Machine Learning Fine-Tuning & Model Evaluation when the work is specifically about lora fine-tuning, dataset cleaning, loss curves, and evaluation benchmarks.
• Load it before a long session that will touch this surface more than once. A one-line edit does not need the full playbook.
• Use it on production SaaS work (multi-tenant apps, paid features, anything that will be reviewed) rather than a throwaway prototype.

When not to use it

• Do not treat the skill as a replacement for a slash command. Skills teach. Commands run a pipeline.
• Do not paste the whole SKILL.md into chat. If Cursor is not loading it, fix the description or invoke the matching command.
• Do not use it as a generic 'write better code' rule. Scope is this domain only.

Example workflow

1. Install the Engineering Kit so .cursor/skills/ml-training lands in the repo.
2. Open a new Cursor agent chat pointed at the files this skill governs.
3. Ask the Machine Learning Engineer (ml-engineer) to read the skill, or run /ml-finetune: ML Model Fine-Tuning & Dataset Pipeline.
4. Review the first artifact against the practices below. If it violates one, stop and correct the file. Do not prompt 'just finish it'.
5. Commit the skill-guided files with the rest of the slice so the next session inherits the same standard.

Example usage

Example prompt: "Read .cursor/skills/ml-training and apply it to this change. Do not invent extra conventions."

Or trigger the pipeline: /ml-finetune: ML Model Fine-Tuning & Dataset Pipeline. That command is written to load this skill.

Check the output against: Deduplicate and clean training datasets to eliminate biased or low-quality pairs

Example output

• The skill itself does not write a single output file. It changes what the agent is allowed to produce in code, schema, and tests.
• When you run /ml-finetune: ML Model Fine-Tuning & Dataset Pipeline, expect repo files plus notes under docs/, not a chat-only answer.

Best practices

• Deduplicate and clean training datasets to eliminate biased or low-quality pairs
• Format training examples with consistent system prompts and conversation turns
• Evaluate fine-tuned models against an un-seen validation dataset to detect overfitting
• Compare model output embeddings against ground truth benchmark answers

Common mistakes

• Ignoring "Deduplicate and clean training datasets to eliminate biased or low-quality pairs" and hoping a later prompt will clean it up.
• Copying the skill into .cursor/rules as always-on. That burns context and fights Cursor's load-on-match design.
• Running two overlapping skills that contradict each other in the same turn.
• Letting the agent skip tests or types because 'the skill is about architecture'.