Usage

Run from the project root directory.

# Full pipeline: mine → generate → filter → evaluate
uv run pytifex

# Target a specific number of disagreements
uv run pytifex --num-examples 10

# Use a more capable Gemini model
uv run pytifex --model gemini-2.5-pro

# Use a local model via llama-server (OpenAI-compatible API)
uv run pytifex --model qwen3-6b --openai-base-url http://localhost:8080/v1

# Use the real OpenAI API
uv run pytifex --model gpt-4o --openai-base-url https://api.openai.com/v1

# Skip GitHub seed fetching
uv run pytifex --no-github

# Verbose output
uv run pytifex -v

Commands

Command	Description
uv run pytifex	Full pipeline (generate + evaluate)
uv run pytifex generate	Generate disagreements only
uv run pytifex check	Run type checkers on existing examples
uv run pytifex eval	Evaluate existing results

Options

Option	Default	Description
--num-examples N	5	Target number of disagreements to find
--batch-size N	15	Examples to generate per LLM batch
--max-attempts N	5	Maximum generation attempts
--max-refinements N	2	Refinement attempts per non-divergent example
--model MODEL	gemini-2.5-flash	LLM model name (see LLM Providers below)
--openai-base-url URL	—	Base URL for an OpenAI-compatible API endpoint
--eval-method METHOD	comprehensive	Evaluation method
--no-github	—	Skip fetching seeds from GitHub issues
-v, --verbose	—	Show all examples, not just disagreements

LLM Providers

Pytifex supports three LLM providers. The provider is selected automatically based on --model and --openai-base-url.

Google Gemini (default)

Requires GEMINI_API_KEY.

export GEMINI_API_KEY=your_key
uv run pytifex --model gemini-2.5-flash    # default
uv run pytifex --model gemini-2.5-pro
uv run pytifex --model gemini-2.5-flash-lite

Cohere

Requires COHERE_API_KEY (or CO_API_KEY). Model names starting with command are automatically routed to Cohere.

export COHERE_API_KEY=your_key
uv run pytifex --model command-a-03-2025
uv run pytifex --model command-r-plus-08-2024

OpenAI-compatible API

Activated by passing --openai-base-url. Works with the real OpenAI API, local models via llama-server, Ollama, vLLM, and any other server that implements the /v1/chat/completions endpoint.

# Real OpenAI API
export OPENAI_API_KEY=your_key
uv run pytifex --model gpt-4o --openai-base-url https://api.openai.com/v1

# Local model via llama-server (no API key needed)
uv run pytifex --model qwen3-6b --openai-base-url http://localhost:8080/v1

# Ollama
uv run pytifex --model llama3 --openai-base-url http://localhost:11434/v1

OPENAI_API_KEY is read from the environment when set; for local servers that don’t require authentication it can be left unset.

Example Output

============================================================
PYTIFEX - Full Pipeline (with disagreement filtering)
============================================================

[STEP 1/2] Generating examples with disagreement filtering...
Target: 2 disagreement examples
Using model: gemini-2.5-flash

[Attempt 1/5] Generating batch of 15...
  Using 5 GitHub issue seeds
  Parsed 19 examples, running type checkers...
  ✓ generic-typevar-bound: DISAGREEMENT {'mypy': 'ok', 'pyrefly': 'ok', 'zuban': 'error', 'ty': 'ok'}
  ✓ self-in-protocol:      DISAGREEMENT {'mypy': 'error', 'pyrefly': 'error', 'zuban': 'error', 'ty': 'ok'}
  Progress: 2/2 disagreements found

GENERATION COMPLETE: 2 disagreements from 19 total examples

Troubleshooting

No disagreements found — Increase --max-attempts or --batch-size, or try --model gemini-2.5-pro.

GitHub rate limit errors — Set GITHUB_TOKEN, or use --no-github.

Type checker not found — Use uv run (auto-installs), or manually pip install mypy pyrefly zuban ty.

Local model not responding — Check that your llama-server process is running and the port matches --openai-base-url. The model name passed to --model must match whatever name the server expects (check your server’s /v1/models endpoint if unsure).