12  Experiment Logging

12.1 Log to directory

You can save any rtemis supervised learning model to disk by specifying an output directory using the outdir argument:

iris.cart <- s_CART(iris,
                    outdir = "./Results/iris_CART")

This will save:

  • A text .log file with the console output
  • A PDF with True vs. Fitted for Regression and a confusion matrix for Classification
  • A RDS file with the trained model (i.e. the R6 object iris.cart in the above example)

The RDS files can be shared with others and loaded back into R at any time.

When running a series of experiments it makes sense to use the outdir argument to save models to disk for reference.

12.2 Interactive logging

The above method of specifying an outdir is the main way to save models to disk. In practice, we often train a series of models interactively and would like to keep track of what we have tried and how it worked out. rtemis includes rtModLogger to help with that. You first create a new logger object, think of it as a container that will hold model parameters and error metrics - not the model itself. Once the logger is created you can add any models to it:

Some synthetic data:

x <- rnormmat(400, 400, seed = 2019)
w <- rnorm(400)
y <- c(x %*% w + rnorm(400))

dat <- data.frame(x, y)
res <- resample(dat)
05-20-25 07:17:32 Input contains more than one columns; will stratify on last :resample
.:Resampling Parameters
    n.resamples: 10 
      resampler: strat.sub 
   stratify.var: y 
        train.p: 0.75 
   strat.n.bins: 4 
05-20-25 07:17:32 Created 10 stratified subsamples :resample


dat.train <- dat[res$Subsample_1, ]
dat.test <- dat[-res$Subsample_1, ]

Initialize a new logger object:

logger <- rtModLogger$new()
logger
.:.:rtemis Supervised Model Logger

   Contents: no models yet

12.2.1 Train some models and add them to the logger:

mod.ridge <- s_GLMNET(dat.train, dat.test,
                      alpha = 0, lambda = .01, verbose = F)
logger$add(mod.ridge)
05-20-25 07:17:33 Added 1 model to logger; 1 total :logger$add


mod.lasso <- s_GLMNET(dat.train, dat.test,
                      alpha = 1, lambda = .01, verbose = F)
logger$add(mod.lasso)
05-20-25 07:17:33 Added 1 model to logger; 2 total :logger$add


mod.elnet <- s_GLMNET(dat.train, dat.test,
                      alpha = .5, lambda = .01, verbose = F)
logger$add(mod.elnet)
05-20-25 07:17:33 Added 1 model to logger; 3 total :logger$add

12.2.2 Plot model performance:

logger$plot(names = c("Ridge", "LASSO", "Elastic Net"))

12.2.3 Get a quick summary:

results <- logger$summary()
results
         Train Rsq  Test Rsq
GLMNET_1 0.9999773 0.5517121
GLMNET_2 0.9998142 0.7465338
GLMNET_3 0.9999467 0.7419457
attr(,"metric")
[1] "Rsq"

12.2.4 Write model hyperparameters and performance to a multi-sheet XLSX file:

logger$tabulate(filename = "./Results/model_metrics.xlsx")

In this example, the XLSX file will contain 3 sheets, one per model. We can save the output of tabulate to a list as well:

tbl <- logger$tabulate()
tbl
$GLMNET_1
  ModelName lambda alpha  Train.MAE  Train.MSE Train.RMSE Train.NRMSE   Train.r
1    GLMNET   0.01     0 0.07712676 0.01000251  0.1000125 0.000816399 0.9999887
  Train.r.p Train.SSE Train.SSR Train.SST Train.Rsq Train.stderr Test.MAE
1         0  2.990749  131644.9  131701.8 0.9999773    0.1000125  10.6674
  Test.MSE Test.RMSE Test.NRMSE    Test.r     Test.r.p Test.SSE Test.SSR
1 182.2739  13.50089  0.1483299 0.7991484 1.306608e-23 18409.67 49026.38
  Test.SST  Test.Rsq Test.stderr
1 41066.62 0.5517121    13.50089

$GLMNET_2
  ModelName lambda alpha Train.MAE  Train.MSE Train.RMSE Train.NRMSE  Train.r
1    GLMNET   0.01     1 0.2330074 0.08185309  0.2860998 0.002335423 0.999923
  Train.r.p Train.SSE Train.SSR Train.SST Train.Rsq Train.stderr Test.MAE
1         0  24.47407  130198.3  131701.8 0.9998142    0.2860998 7.997417
  Test.MSE Test.RMSE Test.NRMSE    Test.r     Test.r.p Test.SSE Test.SSR
1 103.0594  10.15182  0.1115348 0.8717328 1.945751e-32    10409 40018.68
  Test.SST  Test.Rsq Test.stderr
1 41066.62 0.7465338    10.15182

$GLMNET_3
  ModelName lambda alpha Train.MAE  Train.MSE Train.RMSE Train.NRMSE   Train.r
1    GLMNET   0.01   0.5 0.1243415 0.02348304  0.1532418 0.001250907 0.9999776
  Train.r.p Train.SSE Train.SSR Train.SST Train.Rsq Train.stderr Test.MAE
1         0  7.021428  130928.9  131701.8 0.9999467    0.1532418 8.200421
  Test.MSE Test.RMSE Test.NRMSE    Test.r     Test.r.p Test.SSE Test.SSR
1 104.9249  10.24329  0.1125397 0.8713408 2.240629e-32 10597.42 41196.62
  Test.SST  Test.Rsq Test.stderr
1 41066.62 0.7419457    10.24329