REPOSITORY > RESULTS

Abstract

This thesis falls within the field of machine learning and deals with predictive modeling,
where the goal is to learn predictive models. We deal with solving the challenging multitarget
prediction (MTP) tasks. Contrary to the classical predictive modeling, where the
goal is to predict the value of a single target, MTP requires predicting values of multiple
targets.
The central premise of MTP is that the target variables are parts of a structure, where
the structure is seen as a set of connected building blocks that individually carry information
but jointly represent a meaningful concept. We focus on solving the tasks where all
target attributes are either continuous (multi-target regression, MTR) or binary (multilabel
classification, MLC).
MTP tasks can be solved either locally or globally. The local approaches decompose
the original MTP problem into a set of single-target problems. The resulting tasks are
then solved by applying existing methods, which do not account for the possible relations
between the target variables. Alternatively, the global approaches assume interrelations
between the target variables and take advantage of this to learn a single model that is able
to predict all target variables simultaneously. In predictive modeling, ensemble models are
often used to obtain a premium predictive performance. An ensemble is a collection of
models, where predictions are combined to obtain an overall prediction.
In this thesis, we extend ensemble methods for MTP by introducing Random Output
Selections (ROS), a method for learning global ensembles of predictive clustering trees
(PCTs) for MTP. ROS learns PCTs for MTP by using only a randomly selected subset
of target variables. With this, ROS constrains the interrelation assumption to only the
selected subset of target variables.
The proposed method was implemented for several types of tree ensembles for MTP.
We have applied the tree ensembles with ROS to MTR as well as MLC tasks and have extensively
evaluated our methods on a variety of benchmark datasets from various domains.
Moreover, a novel approach to combining base model predictions was introduced. While
usually all ensemble members give predictions for all target variables, in ROS ensembles,
for a given target, we can choose to combine only predictions of base predictive models,
learned by using that target variable. Our results show that tree ensembles with ROS
outperform the current state-of-the-art in MTR and perform comparably to it in MLC
tasks.
Ensemble models are the state-of-the-art in MTP, but are generally not considered
interpretable. Consequently, less powerful, but more interpretable models are often used
instead. Fitted rule ensembles (FIRE) for MTR internally use ensembles of PCTs to derive
a large set of candidate rules of which only a small set is retained after optimization. We
extend this approach by using tree ensembles with ROS (FIRE-ROS) to generate candidate
rules. The results show that FIRE-ROS outperforms all known rule-based methods for
MTR.