ETL & Data
Data Pipeline & Automation Infrastructure
A large-scale data engineering stack that unifies official-source ETL, Indonesian macro indicators, and recurring public-data feeds into dashboard-ready outputs.
Engineered reusable web scraping infrastructure using requests, ThreadPoolExecutor, retry logic, rate limiting, checkpoint/resume systems, Base64 decoding, and data validation.
Processed 30,000+ BKI ship registration records and 15,000+ SNPMB school records across 39 provinces into structured CSV and Excel outputs.
Parsed Bank Indonesia SPI PDF/XLS archives (2003–2025), standardizing mixed-frequency daily, monthly, and yearly time series data.
Automated recurring dashboard datasets via Pandas, openpyxl, pdfplumber, and API integrations for credit card, GAIKINDO automotive, and World Bank indicators.
Implemented fault-tolerant scraping patterns including incremental saves, validation rules, and batch processing for production-scale data collection.
Tech stack
PythonPandaspdfplumberopenpyxlRequestsThreadPoolExecutorETLAutomation