I figured this was worth Sharing after coming across some rather odd behaviour during the provisioning of a new SharePoint farm.
Now before I start, I will say that it’s not me that builds & configures the servers.
So We use a fully scripted setup via Powershell for our SharePoint farm provisioning process, it gets a farm configured joins servers to the farm, set’s up our services, configured search topology, get’s everything as we want for our system.., in this particular environment, the other servers all joined the farm without errors, but the Web Application in IIS was only getting created on one of the three WFE servers in the farm, rather than all of them, the server that was getting the IIS site created was the one where the PowerShell script was being run..
I know that from experience, SharePoint creates it’s IIS Web Applications on other servers in the farm via a “Web Application Provisioning” timer job that run’s on each server joined to the farm, so the problem had to be something to do with this job not running on all the servers, usually a web application appears on all WFE simultaneously or at least within a minute or so of issuing the PowerShell commands from one server to create it.
So I check event logs looking for errors in the timer service on the other WFE servers, nothing, I looked into the ULS logs for errors from OWSTIMER.EXE, nothing, I scratched my head and then slept on the problem, the next morning I was none the wiser, but checking the problematic environment I noticed that the missing WebApps had appeared in IIS while I was away over night.
Hmm I put this one down to experience and moved on, thinking nothing more of this until we tried to deploy some custom farm solutions, again via powershell, on the same farm, the WFE server the powershell was running on WSP deployed straight away, DLL’s GAC’ed etc.. on other two servers nothing deployed to GAC, and the Farm Solutions were stuck at the deploying phase…
WSP’s are again deployed to all servers via a timer job process, thinking about the two issues I started to wonder what the commonality between them was.
It was at this point as I switched from server to server via the Remote Desktop Connection Manager that I spotted the problem, I noticed the time on the windows desktop was different between servers, somehow the time zone was mismatched during the server builds between the WFE server the PowerShell scripts were running on and the other two WFE Servers, the other 2 WFE were 2 hours behind, UTC+8, rather than UTC+10.
I adjusted the time zone of all the servers to match and then WSP files started deploying correctly and also IIS web sites were getting created without issue.
So looks to me that job scheduling via the timer service doesn’t use UTC it uses local server time to decide when to run a job, so if your servers have been configured with different time zones, then you will get some strange behaviour from SharePoint timer jobs, a nice little thing to add to the farm server build QA check list (ensure servers are all set to the same time zone :-))
Thanks for reading.