Along with generalization (and robustness of generalization, using confidence measures), in my experience (ECG arrhythmia detection, speech recognition based speech therapy as two examples), datashift / concept drift adaptation are also absolutely critical.