Are there any performance penalizations because of this? The code for the state machine looks very complex and I'm curious about what's the difference in terms of performance between the pyramid model or the flat model.
What are other metrics that can help validate the F1 score? I'm asking because I believe having a small sample size can skew the number or hide a flaw in the classification algorithm it's scoring.
In other words, what else should I ask to validate that a 70% F1-score is better than a 90% F1-score but on a smaller data set?