Report · marked 25 Aug 2026
hiyouga/LlamaFactory
An agent can work here, but it will waste turns finding its footing.
Python · 74,353 stars · 597 files · branch main
67
/100
B
Instructions
12/27Whether the repo tells an agent how to behave before it starts guessing. Losing marks on agent instruction file and instruction quality.Setup
8/20Whether an agent can install the project and get it running without a human. Losing marks on deterministic install and environment config.Verification loop
22/25Whether an agent can check its own work. This is the category that most decides whether agent output is trustworthy. Losing marks on static type checking.Context economy
16/20Whether the repo fits in a context window, or fights it. Losing marks on file sizes fit in context.Navigability
10/10How quickly anything new can orient itself. Full marks.
Higher is better. Accent marks what is costing you most.
Fix these first
01
Instruction quality
Worth fixing: it's under 200 characters, which is closer to a placeholder than instructions; it never states a build, test, or run command; has no headings to structure it; has no code blocks.
02
Deterministic install
Commit the lockfile your package manager generates. Without it, an agent's install and your install are different builds, and 'works on my machine' becomes unfalsifiable.
03
File sizes fit in context
A single file that fills the context window forces an agent to work from fragments, and it will confidently edit code it never saw. Splitting the worst offenders pays for itself immediately.
The full marking
Every deduction below names the file or setting it came from.
Instructions 12/27 C
Whether the repo tells an agent how to behave before it starts guessing.
✗Agent instruction file
9/12
CLAUDE.md is present, but not the vendor-neutral AGENTS.md.
Rename or symlink to AGENTS.md so tools other than one vendor's can read it. Keeping CLAUDE.md alongside it costs nothing.
Also found: Copilot instructions
✗Instruction quality
0/12
Present but too thin to change an agent's behavior.
Worth fixing: it's under 200 characters, which is closer to a placeholder than instructions; it never states a build, test, or run command; has no headings to structure it; has no code blocks.
✓README as an entry point
3/3
README has a clear getting-started section.
Setup 8/20 C
Whether an agent can install the project and get it running without a human.
✗Deterministic install
0/6
No lockfile. An agent installing dependencies may not get what CI got.
Commit the lockfile your package manager generates. Without it, an agent's install and your install are different builds, and 'works on my machine' becomes unfalsifiable.
✓Discoverable commands
6/6
Commands are declared where an agent will look for them.
Makefile
pyproject.toml scripts
scripts/ directory
✗Environment config
2/4
Env vars are mentioned in the README but there's no example file to copy.
Add a .env.example listing every variable with a safe placeholder value. It's the cheapest possible fix and it unblocks the whole first run.
✗Pinned runtime version
0/2
No pinned runtime version.
Add a .nvmrc, .python-version, or equivalent so the agent's toolchain matches yours.
✗Reproducible environment
0/2
No container or devcontainer definition.
A Dockerfile or devcontainer removes an entire class of 'it won't install' failures.
Verification loop 22/25 A
Whether an agent can check its own work. This is the category that most decides whether agent output is trustworthy.
✓Tests exist
8/8
60 test files against 305 source files.
e.g. scripts/api_example/test_image.py, scripts/api_example/test_toolcall.py, src/llamafactory/train/test_utils.py
✓Test command is discoverable
7/7
An agent can find and run `make test`.
✓Continuous integration
4/4
8 GitHub Actions workflows define what "passing" means.
✓Lint and format rules
3/3
pyproject.toml encodes the house style.
✗Static type checking
0/3
No static type checking.
A type checker gives an agent an error message instead of a runtime surprise. It's the second-fastest feedback loop after the compiler.
Context economy 16/20 A-
Whether the repo fits in a context window, or fights it.
✓No committed build output
6/6
No generated directories committed.
✓.gitignore hygiene
3/3
.gitignore covers 93 patterns.
✗File sizes fit in context
2/6
4 source files are over 100KB.
A single file that fills the context window forces an agent to work from fragments, and it will confidently edit code it never saw. Splitting the worst offenders pays for itself immediately.
src/llamafactory/data/mm_plugin.py — 135KB
src/llamafactory/extras/constants.py — 127KB
src/llamafactory/webui/locales.py — 107KB
✓Repository weight
5/5
About 13.9MB checked out, which is comfortable.
Here is your AGENTS.md
Drafted from what is actually in this repository: the install
command from your lockfile, the commands you already declare, your real directory layout. Anything
marked TODO needs a person. Save it at the root as AGENTS.md.
# AGENTS.md Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024) ## Setup ``` # TODO: the install command. No lockfile was found, so this could not be inferred. ``` ## Commands ``` make check_dirs make ruff_version make build # produce a build make commit make license make quality ``` ## Layout - `src/` 219 source files - `tests/` 37 source files - `scripts/` 21 source files - `tests_v1/` 20 source files - `docs/` 4 source files - `examples/` 4 source files ## Conventions - Tests live alongside the code they cover, following `scripts/api_example/test_image.py`. - CI defines what passing means. See `.github/workflows/docker.yml`, and keep it green. - TODO: add the two or three conventions a newcomer always gets wrong here. ## Gotchas - `src/llamafactory/data/mm_plugin.py` is 135KB. It will not fit comfortably in context, so read it in parts. - No lockfile is committed, so an install here may not match what CI produced. --- Drafted by agentreadme.com from what is in this repository. Everything marked TODO needs a human. Check it in as AGENTS.md at the root.
Open the raw markdown
or
curl -o AGENTS.md agentreadme.com/draft/hiyouga/LlamaFactory.md
Show the mark
The badge re-checks daily, so it keeps up as the repository changes. Use mark again to force it now.
[](https://agentreadme.com/hiyouga/LlamaFactory)Other Python repositories, marked
Think this mark is wrong?
Every deduction above names the file it came from, so this can be settled by looking. If a check missed something, that is a rule worth fixing.