Lab note
Titanic — Machine Learning from Disaster
Kaggle · CatBoost · Feature Engineering · StratifiedKFold
Kaggle Titanic is a classic practice for introductory tabular classification: predicting survival based on passenger features. This project went through a complete ML mini-cycle of research → progressive implementation → submission → wrap-up, ultimately achieving a Public LB of 0.81578 (approximately 341 / 418 correct), landing at the practical upper bound of legitimate solutions.
For the complete methodology and step-by-step lessons, please see the blog post: Kaggle Titanic: From 0.74 to 0.816, Feature Engineering is More Important than Hyperparameter Tuning.
Context
Small-sample tabular binary classification (891 training, 418 test), with the evaluation metric being Accuracy. The goal is not to chase a 1.0 on the leaderboard (mostly cheating by looking up the actual answers), but rather to verify the gap between feature engineering, cross-validation, and the leaderboard under the premise of being reproducible and explainable.
Challenge
- Early Tier 1 features + switching models (RF → CatBoost → voting) could only push the LB to ~0.78, while CV was often higher than the LB.
- Bruteforce OneHot encoding of
Deckor fitting imputation statistics outside of CV would lead to overly optimistic OOF scores and LB drop-offs. - Optuna boosted CV to 0.847, but the Public LB actually dropped to 0.794.
- In Notebook porting, if scaler/encoder were not fitted separately, CV could differ by ~7%.
Solution
Single entry point train.py, modular features_*.py:
data/train.csv, test.csv
→ features_kaggle815.py (Step 5 — 815 notebook features)
→ features_geeky837b.py (Step 7b — strict ipynb porting)
→ CatBoost / RF + StratifiedKFold
→ submission_step_blend.csv (default: average of two survival probabilities)
| Submission File | Method | CV | Public LB |
|---|---|---|---|
submission_step5.csv | 815 notebook FE + CatBoost | 0.824 | 0.81578 |
submission_step7b.csv | Strict Geeky porting + RF | 0.836 | 0.81578 |
submission_step_blend.csv | Average of two probabilities | 0.833 | 0.81578 |
Public demonstration primarily focuses on Step 5; the blend is an ensemble experiment (it changed 6 hard labels compared to Step 5, but the score remained the same—correct and incorrect changes offset each other).
Quick Start
The source code is not currently public. These commands describe the original workflow and require access to the project.
pip install -r requirements.txt
python train.py # Default: blend
python train.py --step 5 # CatBoost single model
python train.py --step 7b # RF strict porting
Impact
- Public LB 0.81578, an improvement of about 5 percentage points over the gender baseline (~0.765).
- The mature feature recipe of Step 5 alone raised the LB from ~0.782 to 0.816 (+3.3%).
- Three independent pipelines achieved the same score on the public leaderboard, verifying that “when the recipe is right, switching the model has a low marginal benefit.”
Key Lessons
- Feature Engineering > Tuning — A mature FE recipe immediately widens the gap; Optuna is prone to overfitting the folds on small data.
- High CV ≠ High LB — The sample split of 891 vs 418 often decouples OOF from the leaderboard.
- Strict Reproduction — Whether scaler/encoder are fitted separately can result in a ~7% difference in CV.
- Knowing When to Stop — 0.81578 is practically the ceiling for honest solutions.
Further Reading
- Code and Documentation: Source code is not publicly available
- Complete Practical Record: Blog Post
- Analogy for Evaluation Decoupling from Offline Metrics: Deployment Simulation and Evaluation Awakening