---

Table of contents

• What is PyMetrics?
• Architecture
• Quick start
• Collected metrics
• CSV outputs
• Dataset pipeline
• Use cases
• Configuration
• Tech stack
• Running tests
• Contributing
• Security
• Citation
• License

---

What is PyMetrics?

PyMetrics is a reproducible, open-source pipeline that:

1. Clones a public GitHub repository at a branch, tag, or pinned commit.
2. Analyzes every .py file with Radon, PyDriller, Pylint, and a custom AST walker.
3. Persists results in SQLite + S3/MinIO, streams live progress over SSE.
4. Exports four ML-ready CSVs and a ZIP bundle per analysis.
5. Builds a curated, multi-repo dataset that can be published on Kaggle for research reuse.

The project targets three audiences:

• 🎓 Researchers who need a reproducible metric pipeline for defect-prediction, code-smell, or software-evolution studies.
• 👷 Engineering teams who want a self-hostable dashboard for code-health monitoring across repos.
• 📊 Data scientists on Kaggle who want a ready-to-use, documented dataset of Python-metric observations.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                      Docker Compose                          │
│                                                              │
│  ┌───────────┐    ┌───────────┐    ┌──────────┐              │
│  │ Frontend  │    │  Backend  │    │  Redis   │              │
│  │  (Nginx)  │───▶│ (FastAPI) │───▶│ (Broker) │              │
│  │  :3000    │    │  :8000    │    │  :6379   │              │
│  └───────────┘    └─────┬─────┘    └────┬─────┘              │
│                         │               │                    │
│                    ┌────┴────┐   ┌──────┴───────┐            │
│                    │ SQLite  │   │Celery Worker │            │
│                    │ (data)  │   └──────┬───────┘            │
│                    └─────────┘          │                    │
│                                  ┌──────┴──────┐             │
│                                  │   Flower    │             │
│                                  │   :5555     │             │
│                                  └─────────────┘             │
│                                                              │
│  Optional: MinIO (:9000/:9001) for S3-compatible storage     │
│  Volumes: ./data/repos, ./data/outputs, ./data/db            │
└──────────────────────────────────────────────────────────────┘

Detailed diagrams and data flow in docs/architecture.md.

Quick start

Option A — Docker Compose (recommended)

git clone https://github.com/cagatayuresin/py-metrics.git
cd py-metrics
cp .env.example .env
docker compose up --build

• Frontend → http://localhost:3000
• Swagger UI → http://localhost:8000/docs
• Flower → http://localhost:5555

Option B — Pull prebuilt images from GHCR

docker pull ghcr.io/cagatayuresin/py-metrics-backend:latest
docker pull ghcr.io/cagatayuresin/py-metrics-frontend:latest
docker compose -f docker-compose.yml up

Option C — Local development

# Backend
cd backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
alembic upgrade head
uvicorn app.main:app --reload

# Frontend (separate shell)
cd frontend
npm install && npm run dev

Collected metrics

1. Raw / Size (6) — radon raw

Metric	Description
loc	Total lines of code
sloc	Source lines of code
lloc	Logical lines of code
cloc	Comment lines of code
blank	Blank lines
comments_ratio	Comment ratio (cloc/loc)

2. Cyclomatic Complexity (9) — radon cc

Metric	Description
cc / cc_mean	Cyclomatic complexity (average)
cc_max, cc_min, cc_sum	CC statistics
cc_rank	Complexity grade (A–F)
num_functions, num_classes, num_methods	Entity counts

3. Maintainability Index (2) — radon mi

Metric	Description
mi_score	Maintainability score (0–100)
mi_rank	Maintainability grade (A/B/C)

4. Halstead (12) — radon hal

Metric	Description
h1, h2	Distinct operators / operands
N1, N2	Total operators / operands
vocabulary, length	Program vocabulary and length
calculated_length	Expected length
volume, difficulty, effort	Information content / difficulty / effort
time, bugs_halstead	Estimated time (s), estimated bug count

5. Object-Oriented / CK (13) — Custom AST walker

Metric	Description
wmc	Weighted Methods per Class
dit	Depth of Inheritance Tree
noc	Number of Children
cbo	Coupling Between Objects
rfc	Response for a Class
lcom	Lack of Cohesion of Methods
nom, nof, nosf, nosm, norm, noc_methods, loc_class	Method / field / override counts

6. Git History (5) — PyDriller

Metric	Description
age_days	File age (days)
revisions	Commit count
authors	Distinct author count
avg_loc_added / avg_loc_deleted	Average lines added/deleted per commit

Git-history columns are populated only when CLONE_SHALLOW=false; the default shallow clone keeps public demos fast.

7. Pylint (5) — pylint

Metric	Description
pylint_score	Code quality score (0–10)
pylint_errors / pylint_warnings	Error / warning counts
pylint_conventions / pylint_refactors	Convention / refactor-suggestion counts

8. Dependency (5) — AST import classification

Metric	Description
imports_count	Total import statements
stdlib_imports, third_party_imports, internal_imports	Classification
fan_out	Number of dependent modules

Full definitions, formulas, and references: dataset/DATA_DICTIONARY.md · docs/metrics.md.

CSV outputs

File	Granularity	Content
metrics_file.csv	File	Raw, CC, MI, Halstead, Git, Pylint, Dependency
metrics_class.csv	Class	CK / OO metrics
metrics_function.csv	Function	CC, Halstead per function
metrics_summary.csv	Project	Aggregated statistics

Per-analysis outputs are written to descriptive directories under data/outputs/, for example:

data/outputs/flask__20260503-002114__ref-main__a3__sha-1a2b3c4d5e6f/

The directory name includes repository, UTC timestamp, requested branch/tag/commit ref, analysis id, and the resolved commit when available. Artifacts are mirrored to S3/MinIO using the same prefix if object storage is configured.

Dataset pipeline

The dataset/ directory contains a reproducible pipeline for collecting a curated public dataset from popular Python repositories. Generated release files are intentionally ignored by Git and should be published through Kaggle or another data-hosting platform instead of being committed to this repository.

The release pipeline produces:

• pymetrics_v1_file.parquet / .csv — per-file metrics.
• pymetrics_v1_class.parquet / .csv — per-class CK metrics.
• pymetrics_v1_function.parquet / .csv — per-function complexity/Halstead.
• pymetrics_v1_summary.parquet / .csv — per-project aggregates.
• DATA_DICTIONARY.md, schema.json, PROVENANCE.md, LICENSE.md.
• Starter notebooks: EDA and Kaggle-ready template.

See dataset/README.md for rebuild commands, schema, provenance, validation, and Kaggle packaging notes.

💡 Use cases

• Defect-prediction models — join the metric tables with an external bug-fix oracle or future-window target.
• Code-quality dashboards — self-host an internal dashboard to track metric trends across your organization's repos.
• Teaching software engineering — materialize abstract concepts (cyclomatic complexity, maintainability index) on real codebases.
• Code review triage — sort PR diffs by complexity delta or CK metric changes.
• Comparative research — benchmark Python frameworks (Django vs. Flask vs. FastAPI) on a common metric footing.
• Refactoring prioritization — quality-gate filters (MI < 20 AND CC_max > 10) surface high-risk files first.

Configuration

Variable	Default	Description
MAX_REPO_SIZE_MB	500	Maximum repository size (MB)
ANALYSIS_TIMEOUT_SEC	600	Analysis timeout (seconds)
SKIP_TEST_FILES	true	Skip test*/ directories during analysis
MAX_FILE_COUNT	5000	Maximum Python files to analyze
BATCH_MAX_REPOSITORIES	1000	Maximum repository URLs accepted by one batch request
CLONE_SHALLOW	true	Use shallow clones for speed; set false for full git-history metrics
PYLINT_ENABLED	true	Toggle Pylint metric extraction
PROCESS_WORKERS	2	Parallel file-processing threads inside one analysis task
CELERY_WORKER_CONCURRENCY	2	Analyses processed concurrently by the worker container
AUTH_ENABLED	false	Enable JWT auth (set true in production)
SECRET_KEY	(required if auth enabled)	JWT signing key
SENTRY_DSN	(empty)	Optional Sentry error tracking
S3_ENDPOINT_URL	(empty)	MinIO/S3 endpoint
CORS_ORIGINS	http://localhost:3000	Allowed CORS origins (comma-separated)

See .env.example for the complete list.

Tech stack

Layer	Technology
Backend	FastAPI, SQLAlchemy 2 (async), Celery, Redis, slowapi, sse-starlette
Frontend	React 18, Vite, Bulma, Recharts, Tabler Icons
Metrics	Radon, PyDriller, Pylint, GitPython, custom AST walker
Infrastructure	Docker Compose, Nginx, Flower, optional MinIO
Database	SQLite (aiosqlite), Alembic migrations
Testing	pytest, pytest-asyncio, pytest-cov, httpx
Docs	MkDocs Material (cagatayuresin.github.io/py-metrics)

Running tests

# Backend, inside the Docker image used in production-like runs
cp .env.example .env
docker compose run --rm backend pytest --cov=app

Frontend build:

docker compose build frontend

🤝 Contributing

We welcome issues, PRs, and dataset contributions! Start here:

• CONTRIBUTING.md — development setup, code style, commit convention
• CODE_OF_CONDUCT.md
• Good first issues

🔒 Security

Please report vulnerabilities privately — see SECURITY.md. Do not open a public issue.

📚 Citation

If you use PyMetrics or the Kaggle dataset in academic work, please cite:

@software{uresin_pymetrics_2026,
  author    = {Uresin, Cagatay},
  title     = {{PyMetrics}: A Python Software Metrics Extraction Platform},
  year      = {2026},
  version   = {0.1.0},
  url       = {https://github.com/cagatayuresin/py-metrics},
  license   = {MIT}
}

See CITATION.cff for machine-readable citation metadata (GitHub auto-renders the "Cite this repository" button).

License

Code is licensed under the MIT License. The Kaggle dataset is released under CC BY 4.0. See per-repo licenses in dataset/PROVENANCE.md.