mirror of
https://github.com/volcengine/verl.git
synced 2025-10-20 13:43:50 +08:00
[data] feat: dump train/test example as JSON (#2666)
### What does this PR do? This PR adds functionality to save one training and one testing example as JSON files for reference, making it easier to inspect dataset formatting and preprocessing. Related to potential future debugging and reproducibility improvements. ### Checklist Before Starting - [ ] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test Manually verified that two files train_example.json and test_example.json are saved correctly in the specified local_dir. ### API and Usage Example This change does not alter the public API. ### Design & Code Changes - Added code to save train_dataset[0] and test_dataset[0] as JSON files in local_dir - Helps with quick inspection and reproducibility of dataset inputs ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [ ] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: easy code - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).
This commit is contained in:
@ -16,6 +16,7 @@ Preprocess the MATH-lighteval dataset to parquet format
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
|
||||
import datasets
|
||||
@ -74,7 +75,13 @@ if __name__ == "__main__":
|
||||
|
||||
train_dataset.to_parquet(os.path.join(local_dir, "train.parquet"))
|
||||
test_dataset.to_parquet(os.path.join(local_dir, "test.parquet"))
|
||||
|
||||
# Save one example as JSON for reference
|
||||
example = train_dataset[0]
|
||||
with open(os.path.join(local_dir, "train_example.json"), "w") as f:
|
||||
json.dump(example, f, indent=2)
|
||||
example = test_dataset[0]
|
||||
with open(os.path.join(local_dir, "test_example.json"), "w") as f:
|
||||
json.dump(example, f, indent=2)
|
||||
if hdfs_dir is not None:
|
||||
makedirs(hdfs_dir)
|
||||
|
||||
|
Reference in New Issue
Block a user