XGBoost, LightGBM, and CatBoost in Practice

XGBoost, LightGBM, and CatBoost build boosted tree ensembles, but they differ in split search, tree structure, categorical processing, regularization, and defaults. Those choices affect the fitted model as well as speed and memory. We will compare explicit configurations, then examine binning, growth constraints, and category handling. A benchmark on one synthetic dataset describes those fits; it cannot establish that the libraries are interchangeable.

A numeric-data comparison

This comparison builds on Gradient Boosting from First Principles. Run the Python blocks in order in an environment with pandas, scikit-learn, XGBoost, LightGBM, and CatBoost installed. The recorded run used scikit-learn 1.9.0, XGBoost 3.4.1, LightGBM 4.7.0, and CatBoost 1.2.10. AUC compares the scores assigned to positive and negative cases; higher values mean better ranking. All examples use held-out rows for comparison. If you use those comparisons to choose settings, reserve a separate test set for the final evaluation.

import pandas as pd
import time
from threadpoolctl import threadpool_limits
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import HistGradientBoostingClassifier
import xgboost as xgb, lightgbm as lgb
from catboost import CatBoostClassifier

X, y = make_classification(n_samples=60_000, n_features=40, n_informative=20,
                           random_state=0)
X = pd.DataFrame(X, columns=[f"f{i}" for i in range(X.shape[1])])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)

def bench(name, build):
    t0 = time.perf_counter()
    with threadpool_limits(limits=4):
        m = build()
    fit = time.perf_counter() - t0
    auc = roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])
    print(f"{name:22s} AUC {auc:.4f}")
    print(f"{name} fit time {fit:.2f} s")

print(f"{'library':22s} AUC")
bench("XGBoost", lambda: xgb.XGBClassifier(
    n_estimators=300, max_depth=6, learning_rate=0.1,
    tree_method="hist", n_jobs=4, random_state=0, verbosity=0).fit(X_tr, y_tr))
bench("LightGBM", lambda: lgb.LGBMClassifier(
    n_estimators=300, num_leaves=63, learning_rate=0.1,
    random_state=0, n_jobs=4, verbosity=-1).fit(X_tr, y_tr))
bench("CatBoost", lambda: CatBoostClassifier(
    iterations=300, depth=6, learning_rate=0.1,
    verbose=0, thread_count=4, random_seed=0, allow_writing_files=False).fit(X_tr, y_tr))
bench("sklearn HistGB", lambda: HistGradientBoostingClassifier(
    max_iter=300, max_depth=6, max_leaf_nodes=63, early_stopping=False,
    learning_rate=0.1, random_state=0).fit(X_tr, y_tr))
# library                AUC
# XGBoost                AUC 0.9927
# XGBoost fit time 1.24 s # varies by machine
# LightGBM               AUC 0.9931
# LightGBM fit time 1.29 s # varies by machine
# CatBoost               AUC 0.9928
# CatBoost fit time 1.90 s # varies by machine
# sklearn HistGB         AUC 0.9925
# sklearn HistGB fit time 1.73 s # varies by machine

The four fits use the same 45,000 training rows, 15,000 held-out rows, and learning rate, with 300 boosting rounds. Histogram Gradient Boosting has internal early stopping disabled so it also uses the requested budget. The depth or leaf limits are roughly comparable, not identical model classes: CatBoost’s symmetric depth-6 tree differs from an unconstrained-depth 63-leaf LightGBM tree. Other regularization and binning defaults remain different. AUC measures ranking here, not probability calibration.

The AUC values range from 0.9925 to 0.9931, a span of 0.0006 on this split. That is a close result for these four configurations; another dataset or tuning budget may separate them. Training uses a four-thread budget, and the recorded times are single-run measurements that vary with the machine and its load. For a practical choice, compare suitably tuned candidates on the validation design for your problem, then consider memory, inference latency, deployment support, and maintenance requirements.

Histogram binning

Histogram methods map numeric feature values into intervals, called bins, and search boundaries between them. For example, values 1.0, 1.1, and 1.2 placed in one bin cannot be separated by that feature’s binned split search. XGBoost’s histogram method defaults to max_bin=256, and LightGBM to max_bin=255. CatBoost specifies the number of borders, with CPU default border_count=254 and different defaults for GPU modes. Borders and bins are related but not identical counts.

For one feature at a node containing \(n_t\) rows and at most \(B\) bins, accumulating gradient and curvature statistics costs roughly \(O(n_t)\), and scanning bin boundaries costs \(O(B)\). Constructing the bin mapping has an upfront cost and may use sampled data. This avoids sorting feature values anew at every node, although exact-split implementations can also reuse sorted information. Histogram subtraction derives one child’s histogram from the parent and the other child, reducing repeated work. Binning, subtraction, data layout, and parallelism all contribute to performance.

Coarser bins change which splits are available. That can lose useful resolution, leave accuracy largely unchanged, or reduce fitting to noise. Lowering the bin limit may reduce memory and runtime, but the effect depends on the data and implementation. Validate it as a modeling choice instead of assuming that it always imposes a small accuracy cost.

Leaf-wise against level-wise growth

XGBoost defaults to grow_policy="depthwise", favoring splits closer to the root, and also supports loss-guided growth. CatBoost’s default SymmetricTree applies the same split condition to every node at a given level; it also offers other growth policies. LightGBM selects the leaf with the largest available split gain. Thus the defaults differ in more than the order in which otherwise identical trees are built.

import pandas as pd
import numpy as np, lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=20_000, n_features=20, n_informative=10,
                           random_state=0)
X = pd.DataFrame(X, columns=[f"f{i}" for i in range(X.shape[1])])
A_tr, A_te, b_tr, b_te = train_test_split(X, y, test_size=0.3, random_state=0)

for nl in (7, 31, 127, 511):
    m = lgb.LGBMClassifier(n_estimators=100, num_leaves=nl, learning_rate=0.1,
                           random_state=0, n_jobs=4, verbosity=-1).fit(A_tr, b_tr)
    depths = []
    def walk(node, d=0):
        if "leaf_value" in node:
            depths.append(d)
            return
        walk(node["left_child"], d + 1)
        walk(node["right_child"], d + 1)
    for t in m.booster_.dump_model()["tree_info"][:20]:
        walk(t["tree_structure"])
    auc = roc_auc_score(b_te, m.predict_proba(A_te)[:, 1])
    print(f"num_leaves {nl:4d}  max leaf depth {max(depths):3d}"
          f"  mean depth {np.mean(depths):5.2f}   AUC {auc:.4f}")
# num_leaves    7  max leaf depth   6  mean depth  3.39   AUC 0.9753
# num_leaves   31  max leaf depth  13  mean depth  6.95   AUC 0.9900
# num_leaves  127  max leaf depth  22  mean depth  9.65   AUC 0.9920
# num_leaves  511  max leaf depth  29  mean depth 12.96   AUC 0.9918

A balanced binary tree with 31 leaves can have maximum depth 5, but leaf count alone does not determine depth. A binary tree with \(L\) leaves can have depth from \(\lceil\log_2 L\rceil\) to \(L-1\). Here the 31-leaf fit reaches depth 13 in the inspected trees, with mean leaf depth 6.95. The code inspects the first 20 of each fit’s 100 trees. Its maximum pools their leaf depths, and its mean weights each inspected leaf equally, not each training row. These structure summaries describe only that subset of trees, while AUC uses the full ensemble.

Long branches can fit localized patterns and also sample-specific detail. A depth-29 branch makes 29 tests, not necessarily tests of 29 distinct features: a feature can recur, and this dataset has only 20 features. Depth alone also does not reveal the number of rows in a leaf. num_leaves, min_data_in_leaf (called min_child_samples in the sklearn wrapper), and max_depth constrain different aspects of growth. The depth measurements do not isolate a cause of faster training or establish overfitting on their own.

A depth cap of 6 permits at most \(2^6=64\) leaves in a binary tree. A leaf cap of 64 permits much deeper unbalanced trees unless depth is also capped. Therefore the relation \(L\leq2^d\) is an upper bound, not a recipe for equivalent fits. LightGBM’s default 31-leaf cap also remains in effect if only max_depth is changed.

Categorical features

A categorical identifier names a group; its numeric ordering may carry no meaning. In this synthetic example each category has a separate random effect on the target. We compare an integer-treated XGBoost baseline with native categorical handling in all three libraries. This is a comparison of configurations, including their different structures and regularization, rather than an isolated test of one encoding algorithm.

import numpy as np, pandas as pd
from sklearn.metrics import roc_auc_score
import xgboost as xgb, lightgbm as lgb
from catboost import CatBoostClassifier

rng = np.random.default_rng(0)
n, K = 40_000, 3000                            # 3000 levels; 10 training rows per level on average
cat = rng.integers(0, K, n)
effect = rng.normal(size=K)
num = rng.normal(size=(n, 4))
logit = effect[cat] + 0.5 * num[:, 0]
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

df = pd.DataFrame(num, columns=[f"n{i}" for i in range(4)])
df["cat"] = cat
tr, te = slice(0, 30_000), slice(30_000, None)

m1 = xgb.XGBClassifier(n_estimators=300, max_depth=6, learning_rate=0.1, tree_method="hist",
                       n_jobs=4, random_state=0, verbosity=0).fit(df.iloc[tr], y[tr])
print(f"XGBoost, category as an integer  AUC "
      f"{roc_auc_score(y[te], m1.predict_proba(df.iloc[te])[:, 1]):.4f}")

dfc = df.copy(); dfc["cat"] = dfc["cat"].astype("category")
mx = xgb.XGBClassifier(n_estimators=300, max_depth=6, learning_rate=0.1,
                        tree_method="hist", enable_categorical=True,
                        n_jobs=4, random_state=0, verbosity=0).fit(dfc.iloc[tr], y[tr])
print(f"XGBoost native categorical       AUC "
      f"{roc_auc_score(y[te], mx.predict_proba(dfc.iloc[te])[:, 1]):.4f}")

m2 = lgb.LGBMClassifier(n_estimators=300, num_leaves=63, learning_rate=0.1, random_state=0, n_jobs=4, verbosity=-1).fit(
    dfc.iloc[tr], y[tr], categorical_feature=["cat"])
print(f"LightGBM native categorical      AUC "
      f"{roc_auc_score(y[te], m2.predict_proba(dfc.iloc[te])[:, 1]):.4f}")

dfs = df.assign(cat=df["cat"].astype(str))
m3 = CatBoostClassifier(iterations=300, depth=6, learning_rate=0.1, verbose=0, thread_count=4, random_seed=0, allow_writing_files=False,
                        cat_features=["cat"]).fit(dfs.iloc[tr], y[tr])
print(f"CatBoost native categorical      AUC "
      f"{roc_auc_score(y[te], m3.predict_proba(dfs.iloc[te])[:, 1]):.4f}")
print("CatBoost boosting type:", m3.get_all_params()["boosting_type"])
# XGBoost, category as an integer  AUC 0.5891
# XGBoost native categorical       AUC 0.6860
# LightGBM native categorical      AUC 0.6715
# CatBoost native categorical      AUC 0.7077
# CatBoost boosting type: Plain

There are 3,000 possible levels and 30,000 training rows, so the average is ten training observations per level, with unequal counts. Treating the ID as numeric lets the tree split contiguous ranges of IDs. It does not force a monotone prediction function, but expressing arbitrary groups of IDs through such ranges can require many splits. With the same depth, round count, and learning rate, switching XGBoost to categorical processing raises AUC from 0.5891 to 0.6860 in this run.

For suitable categorical splits, LightGBM orders categories using node-level gradient and curvature statistics and searches partitions in that order, subject to its categorical constraints; small-cardinality cases can use one-versus-rest splits. CatBoost can construct smoothed target statistics using earlier rows in a permutation, excluding the current row’s own target from that statistic. It also uses other category transformations depending on the configuration. Ordered statistics address self-target contamination, not every possible source of leakage.

A difference between these fits cannot be attributed solely to ordered statistics: they also differ in growth policy, regularization, and categorical split rules. Ordered target statistics and CatBoost’s boosting_type="Ordered" are separate mechanisms. This run prints Plain: categorical statistics are also available in that boosting mode. CatBoost scores 0.7077 here, but attributing that lead to a single mechanism would require a more controlled comparison.

XGBoost introduced categorical support in version 1.5 and partition-based categorical splits in 1.6. Current histogram training can use categorical columns with enable_categorical=True, as shown here. Merely storing identifiers in an integer column does not request this behavior. The categorical-data documentation covers the supported input types and model-saving requirements.

A tuning order

• Fix a leakage-safe validation split and a metric that reflects the decision.
• Try a small set of learning rates with a generous round limit, selecting the stopping count on validation data. A lower rate may need more rounds and a different tree sequence.
• Tune growth limits and leaf constraints for the chosen implementation: depth, leaf count, minimum observations, or minimum summed curvature are different controls.
• Test row and feature sampling where appropriate, checking how the library activates them. In LightGBM, a row fraction below one needs a positive bagging_freq (or subsample_freq) to enable bagging.
• Consider leaf-value penalties and split constraints alongside tree complexity. Existing defaults may already regularize the model; min_child_weight constrains summed curvature in XGBoost, rather than adding an L2 penalty.
• Keep the configuration selected by validation unless a further validated comparison supports changing it. Evaluate the final choice once on data held outside the search.

This is a workable search sequence, not a separation into independent parameter groups. Learning rate, round count, growth limits, and sampling interact. A staged search can save budget, but promising combinations may require revisiting earlier choices. Parameter names and default values should be checked against the library and version actually used.

Exercises

1. What max_bin costs. Vary the histogram bin count from 8 to 255 and report held-out AUC and fit time.

Compare ranking performance and runtime across the five bin limits; identify the trade-off in this run.

import pandas as pd
import time
import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=80_000, n_features=30, n_informative=15,
                           random_state=0)
X = pd.DataFrame(X, columns=[f"f{i}" for i in range(X.shape[1])])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)

for bins in (8, 16, 63, 127, 255):
    t0 = time.perf_counter()
    m = lgb.LGBMClassifier(n_estimators=300, num_leaves=63, max_bin=bins,
                           random_state=0, n_jobs=4, verbosity=-1).fit(X_tr, y_tr)
    fit = time.perf_counter() - t0
    auc = roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])
    print(f"max_bin {bins:4d}   AUC {auc:.4f}")
    print(f"max_bin {bins:4d} fit time {fit:.2f} s")
# max_bin    8   AUC 0.9932
# max_bin    8 fit time 0.70 s # varies by machine
# max_bin   16   AUC 0.9936
# max_bin   16 fit time 0.83 s # varies by machine
# max_bin   63   AUC 0.9937
# max_bin   63 fit time 0.84 s # varies by machine
# max_bin  127   AUC 0.9939
# max_bin  127 fit time 0.97 s # varies by machine
# max_bin  255   AUC 0.9937
# max_bin  255 fit time 1.25 s # varies by machine

2. Porting a configuration between libraries. Compare a fixed XGBoost depth-6 configuration with LightGBM settings that change depth, leaf count, and minimum leaf observations. These candidates are illustrative settings, not the results of a tuning search.

Check which constraints each configuration actually imposes before comparing its AUC.

import pandas as pd
import xgboost as xgb, lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=30_000, n_features=25, n_informative=12,
                           random_state=0)
X = pd.DataFrame(X, columns=[f"f{i}" for i in range(X.shape[1])])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
auc = lambda m: roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])

x = xgb.XGBClassifier(n_estimators=300, max_depth=6, learning_rate=0.1,
                      tree_method="hist", n_jobs=4, random_state=0, verbosity=0).fit(X_tr, y_tr)
print(f"XGBoost max_depth=6                     AUC {auc(x):.4f}")

for label, kw in (("LightGBM max_depth=6 only", dict(max_depth=6)),
                  ("LightGBM num_leaves=64",    dict(num_leaves=64)),
                  ("LightGBM depth 6 + 64 leaves",
                   dict(max_depth=6, num_leaves=64)),
                  ("LightGBM num_leaves=64, min_data=100",
                   dict(num_leaves=64, min_child_samples=100))):
    m = lgb.LGBMClassifier(n_estimators=300, learning_rate=0.1, random_state=0, n_jobs=4,
                           verbosity=-1, **kw).fit(X_tr, y_tr)
    print(f"{label:39s} AUC {auc(m):.4f}")
# XGBoost max_depth=6                     AUC 0.9916
# LightGBM max_depth=6 only               AUC 0.9914
# LightGBM num_leaves=64                  AUC 0.9925
# LightGBM depth 6 + 64 leaves            AUC 0.9916
# LightGBM num_leaves=64, min_data=100    AUC 0.9923

3. Category counts and target encoding. Sweep from 10 to 5,000 levels and compare full-training target means, cross-fitted target means, LightGBM’s native categories, and CatBoost. Use the same training and held-out rows within each cardinality setting.

Compare the methods as the average training count per category declines, including the two target-encoding procedures.

import numpy as np, pandas as pd
import lightgbm as lgb
from catboost import CatBoostClassifier
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import StratifiedKFold

rng = np.random.default_rng(0)
n = 30_000
for K in (10, 100, 1000, 5000):
    cat = rng.integers(0, K, n)
    effect = rng.normal(size=K)
    num = rng.normal(size=(n, 3))
    y = (rng.random(n) < 1 / (1 + np.exp(-(effect[cat] + 0.5 * num[:, 0])))).astype(int)
    df = pd.DataFrame(num, columns=list("abc"))
    tr, te = slice(0, 22_000), slice(22_000, None)

    leaky = df.copy()                          # target encoding on all training rows
    means = pd.Series(y[tr]).groupby(cat[tr]).mean()
    leaky["cat"] = pd.Series(cat).map(means).fillna(y[tr].mean()).values
    m_leak = lgb.LGBMClassifier(n_estimators=200, learning_rate=0.1, random_state=0, n_jobs=4, verbosity=-1).fit(leaky.iloc[tr], y[tr])

    enc = TargetEncoder(target_type="binary", smooth=0.0,
                        cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0))
    cat_train = cat[tr].astype(str).reshape(-1, 1)
    cat_test = cat[te].astype(str).reshape(-1, 1)
    encoded = np.concatenate([enc.fit_transform(cat_train, y[tr]).ravel(),
                              enc.transform(cat_test).ravel()])
    cross = df.assign(cat=encoded)
    m_cross = lgb.LGBMClassifier(n_estimators=200, learning_rate=0.1,
                                 random_state=0, n_jobs=4, verbosity=-1).fit(
        cross.iloc[tr], y[tr])

    dfc = df.copy(); dfc["cat"] = pd.Categorical(cat)
    m_lgb = lgb.LGBMClassifier(n_estimators=200, learning_rate=0.1, random_state=0, n_jobs=4, verbosity=-1).fit(
        dfc.iloc[tr], y[tr], categorical_feature=["cat"])

    dfs = df.assign(cat=cat.astype(str))
    m_cb = CatBoostClassifier(iterations=200, depth=6, learning_rate=0.1, verbose=0, thread_count=4, random_seed=0, allow_writing_files=False,
                              cat_features=["cat"]).fit(dfs.iloc[tr], y[tr])

    a = roc_auc_score(y[te], m_leak.predict_proba(leaky.iloc[te])[:, 1])
    b = roc_auc_score(y[te], m_lgb.predict_proba(dfc.iloc[te])[:, 1])
    c = roc_auc_score(y[te], m_cb.predict_proba(dfs.iloc[te])[:, 1])
    d = roc_auc_score(y[te], m_cross.predict_proba(cross.iloc[te])[:, 1])
    print(f"K={K:5d}  full-mean {a:.4f}  cross-fit {d:.4f}   LightGBM {b:.4f}   CatBoost {c:.4f}")
# K=   10  full-mean 0.6877  cross-fit 0.6891   LightGBM 0.6890   CatBoost 0.6979
# K=  100  full-mean 0.7482  cross-fit 0.7487   LightGBM 0.7408   CatBoost 0.7567
# K= 1000  full-mean 0.7242  cross-fit 0.7190   LightGBM 0.7149   CatBoost 0.7308
# K= 5000  full-mean 0.6620  cross-fit 0.6678   LightGBM 0.5996   CatBoost 0.6834

References

• Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. KDD.
• Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NIPS.
• Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. NeurIPS.