Remove highly correlated variables — filter_high

Identifies and removes numeric columns that have an absolute Pearson correlation above a given threshold. Pairs are evaluated on the subset of rows where both columns are non-missing.

Usage

filter_high_cor(data, cols = NULL, cutoff = 0.9,
                method = "pearson", keep = "first",
                verbose = FALSE)

Arguments

data: A data frame.
cols: Column indices or names of numeric variables to check. If NULL, all numeric columns are used.
cutoff: Absolute correlation threshold. Variables with correlation > cutoff are candidates for removal.
method: Correlation method. Only "pearson" is currently implemented.
keep: Strategy for keeping one variable from each correlated pair: "first" keeps the first occurrence; "highest_var" keeps the one with higher variance.
verbose: Logical; if TRUE, prints removed column names.

Details

The function computes the pairwise-complete Pearson correlation matrix for the selected numeric columns. For each pair with absolute correlation above cutoff, the column to remove is determined by the keep strategy.

Value

A data frame with highly correlated variables removed.

Examples

n <- 100
x1 <- rnorm(n)
x2 <- rnorm(n)
x3 <- x1 * 0.9 + rnorm(n, sd = 0.2)
x4 <- x1 * 0.2 + rnorm(n)
df <- data.frame(x1, x2, x3, x4)
filter_high_cor(df, cutoff = 0.8)
#>               x1          x2           x4
#> 1    1.134965089  0.24189590  0.568241417
#> 2    1.111931845 -1.13275941  1.538553532
#> 3   -0.870777634  1.48990741 -1.133932010
#> 4    0.210731585 -0.24824710 -1.163428894
#> 5    0.069395647  0.18358371  1.581452188
#> 6   -1.662648853  0.40487101 -0.107243971
#> 7    0.810839980 -0.99412447 -0.760242661
#> 8   -1.912345796 -1.08542933 -1.456241570
#> 9   -1.246753429 -0.04854255 -0.801708971
#> 10   0.998154445  0.57608560  0.790092290
#> 11  -0.540872745  0.07383053 -0.635449960
#> 12  -0.216375791  0.70594557  1.298571006
#> 13  -1.621937293  0.33498010 -0.080542830
#> 14  -1.450963965  0.54538781 -0.094312004
#> 15   0.350909731 -1.40290591  0.054662218
#> 16  -0.174546929  0.67705389 -0.332486714
#> 17  -0.591428470 -0.78980045 -0.285964841
#> 18  -1.334027261 -0.46572889  0.889673128
#> 19  -1.097298501 -0.10485207 -1.558223709
#> 20   2.036103609 -1.64785109  1.493989497
#> 21  -0.326489593 -0.09953695 -0.415982115
#> 22   0.774005212 -0.43985764  0.889524287
#> 23   0.785006401 -0.71851145  0.098586796
#> 24   0.763246080 -0.55459760 -1.193163688
#> 25   0.294808760  1.24548918  0.405867021
#> 26  -1.252355924 -1.25892135 -0.888875832
#> 27  -1.009503753 -0.21538448  0.922491604
#> 28   0.751391195 -2.47196171 -0.677454792
#> 29  -1.308353513 -0.67416932 -2.049450840
#> 30   0.527540097 -0.50129719 -0.748501983
#> 31  -0.533539574  1.54232579 -0.678730637
#> 32  -0.398376014 -0.96201807 -0.905605882
#> 33  -0.789569450 -0.87217954 -0.395497984
#> 34  -0.230141136 -1.39762962 -0.641354723
#> 35   0.877184842  0.17980517 -0.235305226
#> 36   0.453733178  1.15409199  0.418738074
#> 37  -0.232464148 -1.19853361 -1.210139441
#> 38   0.870005525 -0.42572440  0.934178770
#> 39   1.656003734  1.36630861  0.328644911
#> 40  -0.006368929 -0.68429739 -0.039501557
#> 41   0.470489453  0.68551221 -1.942139529
#> 42   0.278218649  0.38950354 -0.245585350
#> 43  -0.977902941 -1.30539592 -0.751226693
#> 44  -0.926586142  1.21688801  0.661978187
#> 45   1.919770463  0.79517402  1.774774604
#> 46   0.881277788 -0.48820251  1.407501773
#> 47   0.742081772 -0.90399345 -0.964921108
#> 48   0.147573404 -0.39041842 -0.766539212
#> 49   0.485388565  0.81406342  0.007557836
#> 50   0.151856040 -0.56424928 -1.029285680
#> 51   0.041998754 -1.87420532 -1.595912443
#> 52   0.223422312 -0.14290471  0.836035128
#> 53  -1.010465086  0.77190401 -0.133936489
#> 54   2.401222102 -1.15911793  1.094640420
#> 55   0.801961790 -0.23791553 -1.042675395
#> 56  -0.251207959 -1.22219333 -0.390696427
#> 57   1.212889371  0.11680621 -0.928670760
#> 58  -0.627258086 -0.13249963 -0.954866385
#> 59   1.711158507 -0.03368478  0.571443743
#> 60  -0.394373553 -0.62232642 -0.893867629
#> 61  -2.321490856 -0.70936346 -1.457743686
#> 62   1.364119195  0.87144541 -1.033344620
#> 63   1.132229133  0.10513802 -1.207775945
#> 64  -0.774316319 -0.18693527 -1.017711197
#> 65  -1.410374966 -3.21318853  1.421724211
#> 66  -1.834527581 -1.27561870 -1.022538113
#> 67  -0.269013538  0.76290632 -1.166971333
#> 68  -1.833928577 -0.40681514 -1.772291162
#> 69  -0.814468019 -1.20831778 -0.313537291
#> 70   0.163572122 -0.43932266 -0.361292488
#> 71   0.855519222 -0.37558075  0.423173164
#> 72  -0.819963127 -0.50144240 -0.446699063
#> 73  -0.123602760  0.49661258  0.842947061
#> 74   0.254948236  1.52088094  0.771080768
#> 75   1.718926338  0.98809183  1.399854869
#> 76  -0.958543528  1.24612253  0.051793355
#> 77  -1.604310262 -0.32986733  0.793500610
#> 78  -1.845609422  0.84434471 -0.445674598
#> 79   0.555737185 -0.98107576  1.044072491
#> 80  -0.060119191 -0.13922141 -0.170005577
#> 81   0.772086304  2.18543791 -0.527092254
#> 82  -0.140839387 -0.01282801 -1.248268969
#> 83   0.393093926 -0.30530893 -0.630294207
#> 84   0.224218574 -0.58421346 -0.703835883
#> 85   0.023541985  0.77126850  0.580785923
#> 86  -0.622962660  2.10618936 -0.176691996
#> 87   1.262009381  0.41215704 -0.377935297
#> 88  -0.405774043 -0.26126488 -0.979490874
#> 89   0.666763771  2.07378365  1.646038630
#> 90   0.164639155 -0.77883022 -0.203821462
#> 91   1.781524475  1.13153251 -0.892498527
#> 92   0.711213964 -0.42134513 -0.287766544
#> 93  -0.337691156 -1.02174737  0.025761205
#> 94  -0.009148952  1.21830538  0.831211298
#> 95  -0.125309208 -1.79976067 -0.109994320
#> 96  -2.090846097 -0.30824994 -0.306316180
#> 97   1.697393895  0.01551524 -0.699957164
#> 98   1.063881154 -0.44231772  1.009199044
#> 99  -0.766616636 -1.63800773 -0.053419951
#> 100  0.382007559 -0.64140116  0.813361265