Arcascope · Sleep classification

Is your sleep algorithm good, or are you just testing on a lot of easy-to-classify wake?

scroll

What "easy-to-classify wake" means

Easy-to-classify wake is just what it sounds like: wake that no classifier would ever mess up because it's so obvious. Think lots of visible movement, over a sustained period of time.

A sleep classifier can look artificially good on a dataset if that dataset has a lot of easy-to-classify wake. The score is high, but not because the model is better. The data is just easier.

Try it yourself

Add easy wake, watch the score climb

Below is one synthetic night, scored by a movement-is-wake classifier. Drag the slider to prepend easy-to-classify (ETC) wake (a block of obvious, sustained movement) to the front of the record. The classifier never changes; only the amount of trivial wake does.

The record the classifier is scored on
The score it earns
0.000 AUROC
baseline
0% of the record is trivially-classifiable wake

So what do you do about it?

1

Use benchmark datasets, not new collections.

A shared benchmark lets you separate two things: how much of a score is your classifier being good, and how much is just a property of your test set.

2

Run a simple classifier first.

Before crediting the architecture, see what a trivial baseline scores on the same data. Your complex algorithm might not be doing that much better.

Interactive built on the construction in etc_demo.py: an 8-hour synthetic night with a block of obvious wake prepended, scored by AUROC. Numbers are illustrative.