at the end of the day, friend or not, merlin is a servant.
gwen would not have been at the table either before becoming queen, even though she advised arthur plenty of times before
bbc merlin's worst crime was that merlin never sat at the round table with the others. even without them knowing about his magic, it was obvious he should have been there.