This is actually a brilliant observation that deserves a proper answer. You are not wrong about what you are seeing. But what you are describing is exactly how languages disappear without anyone noticing.
Adamawa alone has over 40 documented languages. Bura, Vere, Chamba, Gaanda, Lala, Bacchama, Bata, Marghi and more and no they are not variations as you pointed out.
But most of them are slowly being swallowed by Hausa and Fulani because those are the languages of trade, mobility and survival.
So yes, your Borno security guard speaks Shuwa Arabic and your Sokoto okada man speaks Hausa and they understand each other perfectly. That does not mean only one language exists. It means one language won the economic argument. This is what linguists call language assimilation. The dominant language does not erase the others overnight. It just makes them less useful for daily survival until the younger generation stops learning them entirely.
Now here are the facts. Ethnologue, which is the world's most authoritative database on languages, currently documents 520 living indigenous languages in Nigeria alone. Not dialects. Languages. Nigeria has also already lost 12 indigenous languages or more to extinction. Gone forever.
The Middle Belt is where this becomes undeniable. Plateau State alone has over 50 distinct languages. Keyword "Dinstinct".
Benue has Tiv, Idoma, Igede and more. Taraba has communities that cannot understand their neighbours two villages away without a translator. Your Yoruba example actually proves the point perfectly. The fact that a Yoruba person can move across the Southwest and be understood is evidence of one dominant language absorbing regional variations over centuries. That process happened. It is still happening everywhere else in Nigeria right now.
Now I am willing to bet you have never heard of Hyam, Ngas, Mwaghavul, Berom, Amo, Buji, Sura, Anaguta, or Irigwe from Plateau State. Or Kilba, Huba, Bura-Pabir, and Chibok from Borno. Or Mumuye, Jenjo, Yukuben, and Wurkum from Taraba. Or Tur, Nyandang, Kugama and Taram further into the riverine communities nobody talks about. Or what about Igala, Ebira, Bassange, Bassa-Nge, Kakanda and Oworo from Kogi alone. I have not even touched Rivers, Cross River, Bayelsa, Edo, Ondo, or Nasarawa yet. You want to know exactly where each of these is spoken? You will have to tour Nigeria for that. And I promise you, this country will humble you in ways no map ever could. The 500 languages are not cap. Most of them are just quietly dying (Bura has an estimated 11,000 speakers with most young Bura people now not able to speak the language) while we debate whether they exist. And that is the real conversation Nigeria should be having.
I don’t believe we speak up to 500 languages in Nigeria. Different dialects ?, yes. But over 500 languages is total cap.
All the South western states speak Yoruba, diffferent variations ,yh but I doubt there’s no where a Yoruba person will go in southwest and not be able to communicate asides some parts of ondo that speak ijaw or the egun speaking communities in ogun state and Lagos & I’m sure this probably applies to people from the south east too.
My former security guy in abk is from borno (north east), he said they speak a language called “shuwa Arabic” but he communicates well with another bikeman from sokoto. (North west) so where did the over 500 languages come from.