That is a great point! I can certainly think of cases where you might want to go with an LLM instead and
we have definitely experimented with that approach. Here are some reasons why we think TreeSeg is more
suitable for us:
1. A more algorithmic approach allows us to bake certain contraints into the model. As an example you can add
a regularizer to incentivize TreeSeg to split more eagerly when there are large pauses. You can also strictly
enforce minimum and maximum sizes on segments.
2. If you are interested in reproducing a segmentation with slight variations you might not have good results
with an LLM. Our experience has been that there is significant stochasticity in the answers we get from an LLM.
Even if you try to obtain a more deterministic answer (i.e. set temp to zero), you will need an exact copy of
the model to get the same result in the future. Depending on what LLM you are using this might not be possible
(e.g. OpenAI adjusts models frequently). With TreeSeg you only need your block-utterance embeddings, which you
probably have already stored (presumably in a vector db).
3. TreeSeg outputs a binary tree of segments and their sub-segments and so forth... This structure is important
to us for many reasons, some of which are subjects of future posts. One such reason is access to a continuum
between local (i.e. chapters) and global (i.e. full session) context. Obtaining such a hierarchy via an LLM
might not be that straightforward.
4. There is something attractive about not relying on an LLM for everything!
1. A more algorithmic approach allows us to bake certain contraints into the model. As an example you can add a regularizer to incentivize TreeSeg to split more eagerly when there are large pauses. You can also strictly enforce minimum and maximum sizes on segments.
2. If you are interested in reproducing a segmentation with slight variations you might not have good results with an LLM. Our experience has been that there is significant stochasticity in the answers we get from an LLM. Even if you try to obtain a more deterministic answer (i.e. set temp to zero), you will need an exact copy of the model to get the same result in the future. Depending on what LLM you are using this might not be possible (e.g. OpenAI adjusts models frequently). With TreeSeg you only need your block-utterance embeddings, which you probably have already stored (presumably in a vector db).
3. TreeSeg outputs a binary tree of segments and their sub-segments and so forth... This structure is important to us for many reasons, some of which are subjects of future posts. One such reason is access to a continuum between local (i.e. chapters) and global (i.e. full session) context. Obtaining such a hierarchy via an LLM might not be that straightforward.
4. There is something attractive about not relying on an LLM for everything!
Hope this is useful to you!