I think the reason why Durin is less carefree around Albedo compare to Wan is the subconscious pressure. He doesn't want to dissapoint Albedo, he wants to prove himself since their relationship is also of a student and a mentor. With Wan, there are no such expectations to uphold