So, my peer and I are in the process of updating a certificate on an Office Web Apps (OWA) server for SharePoint. The process is not actually that straightforward and we are still working it out. This isn’t really about updating the certificate, though- it’s really about the scary moment that you look in your OWA server and see your IIS Sites gone- totally missing!
I noticed an alert in SCOM and that the OWA apps weren’t working. No one really goes crazy if OWA goes down but a few important people may notice. In my opinion, it’s a really nice feature- a nice-to-have- but editing docs has become a common feature in Office 365, OneDrive, SharePoint, Dropbox, GSuite, etc, so we can’t let it stay offline. I noted that the sites were completely gone in IIS on the OWA server. GONE.
Now, if you’ve used IIS a bit, you understand that the sites don’t just vanish- not without something really crazy going down, like data corruption. You will most likely have a blue screen (or black screen) on your server before IIS just deletes your sites.
So like a good little admin, I jumped into the Event Viewer. I saw the reboot, then was able to trace a bunch of nasty events that started when it went down for patching. So I knew the problem was related to its monthly Windows Updates- or was it? I also noted that the Office Web Apps service was still in place, and would not start. The Certificates will still present in CertMgr.
I started searching to see if there were any known issues for this month’s patching and OWA. Nothing. Plenty of warnings, of course, about not patching your SharePoint server farm with WSUS or SCCM. In fact, SharePoint updates can’t be delivered to WSUS or SCCM anyway. They just don’t happen.
Then I remembered a problem we had a last year when we were updating an RDP certificate on a Server Core system. We were mucking with it but couldn’t make it work. I also remembered that the server stopped responding because of a certificate mismatch. It turned out that we had 2 certificates with the same name in the Personal store, and by removing the duplicate, whether it was the old one or the new one, and rebooting the server, we could fix it.
Sure enough, I saw we had the old and new certificates in the personal store, and they had the same names.
I use Rubrik for backups (which is incredible, BTW) so I can easily mount several versions of the backed up server at once in vSphere. I looked back at versions of the server before and after the server patched. All of the versions for the last month didn’t have the IIS sites- but OWA was working until the reboot.
Then a light bulb went on!
The server didn’t have an issue until the reboot, even though the sites weren’t showing in IIS. They were actually still there.
I exported and deleted the duplicate certificate in the Personal Store (make sure you do this in Computer certificates, not User. It won’t be present in User certificates anyway). Then I was able to start the Office Web Apps service. Web Apps started working right away and the IIS sites reappeared.
So don’t leave duplicate certificates in the Personal Store.