Here is my personal educated opinion about closed source, open source, and chain of trust in a hardware wallet.
It's a long read, but it is very simple, pragmatic, balanced and educational. If you have some interest into hardware wallets and privary key security, please take the time to read it.
A hardware wallet is a small computer with a simple purpose: provide secure entropy (randomness) and manage secrets while making sure these secrets cannot be accessed in any unauthorized way.
To achieve this, you can either use regular electronic chips, or use a secure element which is a microprocessor with advanced security capabilities (reducing side channels attack surface, mitigating hardware attacks such as glitching, etc).
As secure elements come initially from the banking industry which applies "security by obscurity" principles, they come with strong NDAs preventing device manufacturers to reveal APIs and thus publish the source code of the firmware.
So you can either have open source with regular hardware, but with less security, or closed source with secure elements, allowing higher level of security (adding a "root of trust" that I'll explain below).
Compiling an open source firmware is quite easy for anyone with an IT background. However, there is an issue: you must trust the device you are using.
When plugging the HW to flash the firmware that will manage the secrets, you do not know if you are really talking to the microprocessor or a simulated version of it (a more powerful microprocessor with the same physical size and look, but emulating a security theater that will eventually access the secrets in an unauthorized way).
This is called a supply chain attack, that can happen at many stages, including at the manufacturer level.
To mitigate this risk, there is the "holographic seal" but it's not that efficient and doesn't prevent any attack at the manufacturer level, or there is the "root of trust". This is a master key from the manufacturer, implanted in the secure element at the factory level, that can cryptographically prove you are indeed talking to the real chip. This cannot be used with open source as the master key would obviously be revealed during compilation.
This means that using an open source HW, you must trust that the manufacturer will not put a backdoor in the electronics, and using a closed source HW you must trust that the manufacturer will not put a backdoor in the firmware.
The former can be mitigated by assembling yourself the electronics, from components you'll source, and the latter by governance, internal audits and certifications from third party.
The general public doesn't wish to be bothered about all this and just want an "easy solution that I can trust".
If you think the manufacturer, open source or closed source, is your adversary then there cannot be any trust and therefore there is no sense to use the device.
Using a hardware wallet, whatever its conception, requires a level of trust into the manufacturer.
Sure, you could do everything yourself using airgapped computers etc, but that's not a realistic solution for 99.99% of the users, hence the need of hardware wallets.
But then what about a solution where the secure element cannot be updated, accessed in any other way than doing a signature and without possibility of doing anything else? First, no being to update a system us far from ideal, but the reality is that it doesn't change anything. You still need to trust the manufacturer! For instance, the source of randomness generating the private key could have a very low entropy, opening the way to brute force attacks for the one knowing the entropy range.
The only trustless way to manage your keys with a hardware wallet or airgap system is by doing everything from A to Z. Any compromission on the process introduces trust into a third party. It therefore requires not only good understanding of what you are doing, but also huge discipline.
Security is always a matter of trade-offs.
It is important to understand the chain of trust, measure the risks (scenarios where your secrets get unauthorized access) and the rewards (ease of use).
For example: what is the risk that a reputable hardware wallet manufacturer adds a backdoor? What are the internal processes of this manufacturer to prevent a rogue agent attack? What is the "game theory" risk that the manufacturer decides to execute an exit scam? What is the legal risk assessment of a government forcing a backdoor in the firmware, or in the hardware? etc.
But also: what is the risk of managing myself everything with a complex hardware and software system, and doing a critical mistake? What is the risk of someone getting physical access to my hardware wallet and extracting the keys? What is the risk that I update my HW with a rogue firmware because I'm sandboxed on my wifi?
Nothing is simple, nothing is black and white.
What is really important and critical, is to understand how everything works and make an educated opinion.
In the same way that I tell people to study Bitcoin, not because they'll get rich or whatnot, but because they'll understand how money works and it'll change their perspective.