Benchmark quantile-selection methods on correlated designs — benchmark_quantile

benchmark_quantile_selection() runs a reproducible simulation study over a set of scenarios and compares three selectors:

Usage

benchmark_quantile_selection(
  scenarios = default_quantile_benchmark_scenarios(),
  methods = c("lasso", "lasso_tuned", "selectboost"),
  replications = 20,
  threshold = 0.55,
  selection_metric = c("hybrid", "frequency"),
  selectboost_args = list(B = 20, step_num = 0.25, screen = "auto", tune_lambda = "cv",
    lambda_rule = "one_se", lambda_inflation = 1.25, nlambda = 12, folds = 5, repeats =
    1, subsamples = 25, sample_fraction = 0.5, complementary_pairs = TRUE, max_group_size
    = 15, verbose = FALSE),
  tuned_args = list(method = "cv", rule = "one_se", lambda_inflation = 1.25, nlambda =
    12, folds = 5, repeats = 1, verbose = FALSE),
  lasso_args = list(),
  standardize = TRUE,
  eps = 1e-06,
  seed = NULL,
  verbose = interactive()
)

Arguments

scenarios: Named list of scenario specifications. Each entry is passed to simulate_quantile_data(). Use default_quantile_benchmark_scenarios() for a ready-made validation grid.
methods: Methods to benchmark. Supported values are "lasso", "lasso_tuned", and "selectboost".
replications: Number of Monte Carlo replications per scenario.
threshold: Selection-frequency threshold used when extracting the stable support from selectboost_quantile().
selection_metric: Summary score used when extracting the stable support from selectboost_quantile(). "hybrid" combines stability and fitted effect size, while "frequency" uses mean path frequency alone.
selectboost_args: Additional named arguments passed to selectboost_quantile() for the "selectboost" method.
tuned_args: Additional named arguments passed to tune_lambda_quantile() for the "lasso_tuned" method.
lasso_args: Additional named arguments passed to quantile_lasso_selector() for the "lasso" method.
standardize: Should the lasso baselines use the same standardized design as selectboost_quantile()?
eps: Numerical tolerance used to turn coefficients into selections.
seed: Optional random seed.
verbose: Should progress messages be emitted?

Value

An object of class "benchmark_quantile_selection" with raw per-replication results in results.

Details

"lasso": plain quantreg::rq.fit.lasso() support,
"lasso_tuned": quantile lasso with tune_lambda_quantile(),
"selectboost": stable support extracted from selectboost_quantile().

Each row in the returned benchmark table records support recovery, false discoveries, runtime, and failure status for one scenario, replication, and method.

Examples

scenarios <- default_quantile_benchmark_scenarios(
  tau = 0.5,
  regimes = "moderate_corr"
)
bench <- benchmark_quantile_selection(
  scenarios = scenarios,
  replications = 1,
  selectboost_args = list(B = 2, step_num = 1, tune_lambda = "bic", nlambda = 3),
  tuned_args = list(method = "bic", nlambda = 3),
  verbose = FALSE,
  seed = 1
)
summary(bench)
#>                scenario tau      method replications mean_tp mean_fp mean_fn
#> 1 moderate_corr_tau_0_5 0.5       lasso            1       6      10       0
#> 2 moderate_corr_tau_0_5 0.5 lasso_tuned            1       6      11       0
#> 3 moderate_corr_tau_0_5 0.5 selectboost            1       6       0       0
#>   mean_support_size mean_tpr mean_precision  mean_fdr mean_runtime_sec
#> 1                16        1      0.3750000 0.6250000            0.544
#> 2                17        1      0.3529412 0.6470588            0.009
#> 3                 6        1      1.0000000 0.0000000            0.249
#>   sd_runtime_sec failure_rate support_stability
#> 1             NA            0                NA
#> 2             NA            0                NA
#> 3             NA            0                NA