HumanStudy-Bench: Community Edition

Open community-driven expansion of the HumanStudy-Bench benchmark

LLMs are increasingly used to simulate human participants in social science research, but existing evaluations conflate base model capabilities with agent design choices, making it unclear whether results reflect the model or the configuration.

Overview

HumanStudy-Bench treats participant simulation as an agent design problem and provides a standardized testbed — combining an Execution Engine that reconstructs full experimental protocols from published studies and a Benchmark with standardized evaluation metrics — for replaying human-subject experiments end-to-end with alignment evaluation at the level of scientific inference.

How to Contribute a Study

1. Fork and clone

git clone https://github.com/<your-github-id>/HumanStudy-Bench.git
cd HumanStudy-Bench
git checkout -b contrib-<yourgithubid>-013

2. Create your study folder

Add a new directory under studies/ with the required folders:

studies/<yourgithubid>_013/
  ├── index.json
  ├── source/
  ├── scripts/
  └── README.md

See the docs below for what goes inside each folder and the exact schemas:

#	Guide	Description
1	What Should I Submit?	Overview of contribution, required folders and files
2	How to Extract Data from a Paper	Paper hierarchy, AI extraction prompt, walkthrough example
3	How to Build Your Study Files	Schemas, code examples, and contracts for each file
4	How to Submit Your Study	Fork, verify, push, and open a PR

3. Verify locally

bash scripts/verify_study.sh <yourgithubid>_013

4. Commit and push

git add studies/<yourgithubid>_013/
git commit -m "Add study: <Your Study Title>"
git push origin contrib-<yourgithubid>-013

5. Open a Pull Request

Open a PR on GitHub targeting the main branch. Maintainers assign final study_XXX numbering by merge order. CI runs validation automatically; confirmation is by human review.

You can also submit a study via web upload at hs-bench.clawder.ai/contribute.

Existing Studies

The 12 foundational studies (cognition, strategic interaction, social psychology) serve as reference examples. Browse them on the website or locally under studies/.

Citation

If you use HumanStudy-Bench, please cite:

@misc{liu2026humanstudybenchaiagentdesign,
      title={HumanStudy-Bench: Towards AI Agent Design for Participant Simulation},
      author={Xuan Liu and Haoyang Shang and Zizhang Liu and Xinyan Liu and Yunze Xiao and Yiwen Tu and Haojian Jin},
      year={2026},
      eprint={2602.00685},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.00685},
}

License

MIT License. See LICENSE for details.