Report · marked 25 Aug 2026
karpathy/autoresearch
An agent is close to flying blind in this repository.
Python · 94,687 stars · 10 files · branch master
34
/100
D
Instructions
3/27Whether the repo tells an agent how to behave before it starts guessing. Losing marks on agent instruction file and instruction quality.Setup
8/14Whether an agent can install the project and get it running without a human. Losing marks on discoverable commands.Verification loop
0/25Whether an agent can check its own work. This is the category that most decides whether agent output is trustworthy. Losing marks on tests exist and test command is discoverable.Context economy
20/20Whether the repo fits in a context window, or fights it. Full marks.Navigability
2/10How quickly anything new can orient itself. Losing marks on written documentation and repository description.
Higher is better. Accent marks what is costing you most.
Fix these first
01
Agent instruction file
Add an AGENTS.md at the root: what the project is, how to install, how to run, how to test, and the two or three conventions a newcomer always gets wrong.
02
Instruction quality
Instructions earn their keep by naming exact commands. 'Run the tests with `pnpm test`' beats three paragraphs of philosophy.
03
Tests exist
This is the single highest-leverage change on this page. Agents don't need full coverage, they need one command that goes red when they break something. Even a handful of tests on the core path converts blind editing into a feedback loop.
The full marking
Every deduction below names the file or setting it came from.
Instructions 3/27 F
Whether the repo tells an agent how to behave before it starts guessing.
✗Agent instruction file
0/12
None found. Every agent that opens this repo starts from zero.
Add an AGENTS.md at the root: what the project is, how to install, how to run, how to test, and the two or three conventions a newcomer always gets wrong.
Looked for AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, .cursorrules
✗Instruction quality
0/12
Nothing to judge, since there are no instructions.
Instructions earn their keep by naming exact commands. 'Run the tests with `pnpm test`' beats three paragraphs of philosophy.
✓README as an entry point
3/3
README has a clear getting-started section.
Setup 8/14 B-
Whether an agent can install the project and get it running without a human.
✓Deterministic install
6/6
uv.lock pins the dependency tree.
Found uv.lock
✗Discoverable commands
0/6
No declared commands. An agent has to infer how to build and run this from the file tree.
Declare the handful of commands that matter in package.json scripts, a Makefile, or a justfile. Naming them turns guesswork into a lookup.
–Environment config
n/a
Not applicable — this reads as a library, with no environment to configure.
✓Pinned runtime version
2/2
The language runtime version is pinned.
Found .python-version
–Reproducible environment
n/a
Not applicable — a library doesn't need a container to be worked on.
Verification loop 0/25 F
Whether an agent can check its own work. This is the category that most decides whether agent output is trustworthy.
✗Tests exist
0/8
No test files found. An agent has no way to know whether its change broke anything.
This is the single highest-leverage change on this page. Agents don't need full coverage, they need one command that goes red when they break something. Even a handful of tests on the core path converts blind editing into a feedback loop.
✗Test command is discoverable
0/7
There's no obvious way to run the tests. Even if tests exist, an agent won't reliably find the entry point.
Add a `test` script or a `test` target that runs the whole suite with no arguments. The convention is the interface — an agent tries `npm test` and `make test` before it tries reading your CI config.
✗Continuous integration
0/4
No CI config. Nothing outside a developer's laptop enforces correctness.
A CI workflow doubles as machine-readable documentation: it's the one file that states, unambiguously, the exact commands that must succeed.
✗Lint and format rules
0/3
No linter or formatter config. Style is tribal knowledge, so agent output will drift from yours.
A formatter config is the cheapest way to stop reviewing whitespace in agent diffs. It moves style from opinion to a command.
✗Static type checking
0/3
No static type checking.
A type checker gives an agent an error message instead of a runtime surprise. It's the second-fastest feedback loop after the compiler.
Context economy 20/20 A+
Whether the repo fits in a context window, or fights it.
✓No committed build output
6/6
No generated directories committed.
✓.gitignore hygiene
3/3
.gitignore covers 14 patterns.
✓File sizes fit in context
6/6
No source file is large enough to crowd out a context window.
✓Repository weight
5/5
About 530KB checked out, which is comfortable.
Here is your AGENTS.md
Drafted from what is actually in this repository: the install
command from your lockfile, the commands you already declare, your real directory layout. Anything
marked TODO needs a person. Save it at the root as AGENTS.md.
# AGENTS.md AI agents running research on single-GPU nanochat training automatically ## Setup ``` uv sync ``` ## Commands ``` # TODO: nothing declares a build or test command, so an agent has to guess. # This is the single most valuable section of this file. Fill it in. ``` ## Conventions - There are no tests yet. An agent has no way to check its own work here. - TODO: add the two or three conventions a newcomer always gets wrong here. --- Drafted by agentreadme.com from what is in this repository. Everything marked TODO needs a human. Check it in as AGENTS.md at the root.
Open the raw markdown
or
curl -o AGENTS.md agentreadme.com/draft/karpathy/autoresearch.md
Show the mark
The badge re-checks daily, so it keeps up as the repository changes. Use mark again to force it now.
[](https://agentreadme.com/karpathy/autoresearch)Other Python repositories, marked
Think this mark is wrong?
Every deduction above names the file it came from, so this can be settled by looking. If a check missed something, that is a rule worth fixing.