← Explore all skills

MODEL EVALUATION / Hugging Face

Hugging Face Local Model Evaluations

Chooses a local evaluation framework and inference backend, starting with a small smoke run before scaling evaluation work.

“Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.”

01 / THE REASONING

Why this made the selection.

  • Separates local GPU inference from a locally launched client that calls remote Inference Providers.
  • Explicitly keeps remote job orchestration and publication outside this workflow.

02 / THE REVIEW RECORD

What we actually inspected.

Source review has boundaries.
A clear record is more useful than a “safe” badge.

Material inspected

  • skills/huggingface-community-evals/SKILL.md
  • skills/huggingface-community-evals/scripts/inspect_eval_uv.py
  • skills/huggingface-community-evals/examples/USAGE_EXAMPLES.md
  • README.md skills table and installation section
  • LICENSE (Apache License 2.0 text)

Our findings

  • The inspected helper selects the hf-inference-providers backend and forwards HF token aliases to its subprocess environment.
  • It invokes Inspect with a bounded connection setting and surfaces a failing subprocess rather than reporting success.
  • The definition routes remote Jobs and publishing to other workflows and recommends a small sample limit first.
  • Before a local GPU run the SKILL.md checks uv --version, printenv HF_TOKEN and nvidia-smi. Without nvidia-smi it falls back to the provider-backed script.
  • The SKILL.md names a hugging-face-jobs skill for remote runs. The skills/ directory at this commit has no such skill.
  • Re-pinned from abc20ae526d8b4c0e4dff89f904adce28a4a0eb6 to c3ff942be3e07116b475253f0a8fc4f54682c1a8: a git tree comparison shows no change to any file under skills/huggingface-community-evals, to LICENSE or to README.md between the two commits. upstreamDescription now carries the full frontmatter description; the earlier quote stopped mid-sentence.

Not established by this review

  • Source inspection only. No evaluation, GPU check or provider call was run, and the skill was not installed.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

Add a skill to your workflow.

Upstream setup instructions ↗

DOCUMENTED COMMAND

npx skills add huggingface/skills/skills/huggingface-community-evals

Run in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.

Optional: install the reviewed revision

Keeps the original source and installer version where recorded, while leaving your AI tools and installation scope for you to choose.

npx skills add https://github.com/huggingface/skills/tree/c3ff942be3e07116b475253f0a8fc4f54682c1a8/skills/huggingface-community-evals
  1. Install the complete skill directory with its references and helpers; a copied SKILL.md alone may be incomplete.
  2. Choose your AI tools and project or global scope in the installer, then review its permissions before using the skill.

Before you start

  • Node.js/npm for this installer
  • An agent that supports the Agent Skills format
  • Model/dataset access and suitable compute for the chosen backend
  • HF token for gated models or provider access as required

04 / FOLLOW THE EVIDENCE

The source trail.

Our notes are separate from the original resource.
Check upstream before adopting a new version.

  1. https://github.com/huggingface/skills/blob/c3ff942be3e07116b475253f0a8fc4f54682c1a8/skills/huggingface-community-evals/SKILL.md

    Supports: summary, upstreamDescription, whySelected, bestFor, limitations, review, compatibility, access

  2. https://github.com/huggingface/skills/blob/c3ff942be3e07116b475253f0a8fc4f54682c1a8/skills/huggingface-community-evals/scripts/inspect_eval_uv.py

    Supports: review, limitations, access

  3. https://github.com/huggingface/skills/blob/c3ff942be3e07116b475253f0a8fc4f54682c1a8/skills/huggingface-community-evals/examples/USAGE_EXAMPLES.md

    Supports: review, limitations, access

  4. https://github.com/huggingface/skills/blob/c3ff942be3e07116b475253f0a8fc4f54682c1a8/LICENSE

    Supports: license

  5. https://github.com/huggingface/skills/blob/c3ff942be3e07116b475253f0a8fc4f54682c1a8/README.md

    Supports: install, compatibility, access

  6. https://github.com/vercel-labs/skills/blob/7407f3893ad4dceab546ac002c3ef806e4000c73/README.md

    Supports: install

  7. https://github.com/vercel-labs/skills/blob/7407f3893ad4dceab546ac002c3ef806e4000c73/src/git.ts

    Supports: install

Evidence & Ask