Syntk
syntk is a synthetic-data toolkit — generate and augment datasets using LLMs. Written by Markus (Hei) and used across the fleet's training and evaluation pipelines.
Stack: Python. Public repo.
Supports Parquet, CSV, JSON, JSONL, and TSV; any OpenAI-compatible API endpoint (OpenAI, OpenRouter, local llama.cpp / vLLM).
Pipelines
- Column pipeline — fill a dataset column with LLM-generated values from a prompt template that can reference any other column.
- Bootstrap pipeline — grow a tabular dataset by N rows via few-shot LLM prompting.
Resume support (auto-resumes interrupted processing), YAML-configured for reproducibility, and built-in experiment tracking to TensorBoard, MLflow, W&B, or Aim.
Hugging Face URIs
input_file and output_file accept hf://datasets/<owner>/<repo>/<path> URIs. Reads stream straight from the Hub via huggingface_hub. Writes auto-create the dataset repo (private, exist_ok=True) so the first run doesn't FileNotFoundError. Auth uses the same HF_TOKEN the CLI does.
Install
pip install git+https://github.com/marksverdhei/syntk.git
syntk column examples/column_rate_difficulty.yaml
syntk column config.yaml --limit 100 --model "gpt-4"
Sources
See also
- bakery — where synthetic-data prompts get baked into weights.