The bell of the famous clock will fall silent for four years to protect workers’ hearing during a £29m restoration of the Elizabeth Tower, the Great Clock and the Great Bell.
Sir Robert McAlpine secured the £3.5m pre-construction services deal last year.
An HSE spokesperson said: “We’ve been concerned by a minority of the reaction to the announcement relating to the Big Ben conservation project in London.
“People’s health should not be made worse by the work they do, so it is important that no worker should suffer any hearing loss while working on this project. We find any attempt to trivialise this unhelpful.
“As part of our regulatory role, HSE has liaised with both the client and the principal contractor on this major construction project in central London.
“This has been one of many projects where we work with contractors in the planning stages, and we’ve noted how intricate, complex and challenging this particular exercise will be.
“Health and safety aside, we understand these challenges would have silenced Big Ben’s chimes for at least two years anyway.
“While we were aware part of the project related to the clock, we have not been involved in discussions about how that work will be specifically carried out.
“There is broad agreement that the noise risks associated with working around the clock bells are highly significant and we would expect the principal contractor to manage those risks.
“How it does so is a matter for those involved and their client.”