Office Web Apps - Word Viewing Error

All of sudden, I noticed that the Office Web Apps "Word Viewing Service" stopped working.

The authenticated mode is switched from "NTLM" to "Negotiation" recently, maybe that's the reason. However, I did the same change on Development and Test environment, which all work well.

Two days struggling. There are some good posts on Internet such as here and here. But none of them suits my case. I even reinstalled "Word Viewing Service Application" through Central Administration.

I had granted enough rights to Office Web Apps service account (over Web Application and Content Database).  There is no DC installed with SharePoint. No "Document ID" issues. The servers are rebooted a  number of times

What else could you it be?

Luckily, I tried to install all windows OS patches on SharePoint application server (which hosts Office Web Apps).  That fixed the problem!

I don't know which patch fixed it, and how this problem happened. Any way, the clients are happy now :-)

Little tips for testing.

1. Reboot IE before testing changes;
2. Delete the cached files before testing changes.  The cached files are stored in
http://SPServer/sites/Office_Viewing_Service_Cache/cache

Below are the error message I got during the trouble shooting.


=============================================

Error getting cached item Ffd41c4e40bc04ce4bec4c7df5caf2d77md82e0901a5e94bdc8c6900cc7d214fc8mbc12e76fbf5f4bbf826ff55b08480fc5m, p_1_4.zip, System.ServiceModel.ServerTooBusyException: The HTTP service located
 at http://spAppServer:32843/9d312a4445b148eb9df04084fe20495f/Conversion.svc is too busy.  ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.  
 at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)  
 at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)     -
 -- End of inner exception stack trace ---    Server stack trace:    
 at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)  
 at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)  
 at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)  
 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)  
 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown
 at [0]:    
 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)  
 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)  
 at Microsoft.Office.Web.Conversion.Framework.Remoting.IConvertingService.EndGetCachedItem(IAsyncResult asyncResult)  
 at Microsoft.Office.Web.Conversion.Framework.Remoting.ConvertingServiceProxy.OnItemDataReceived(IAsyncResult ar)


=============================================

Requesting item that was not generated StackTrace:
 at Microsoft.Office.Web.Common.Log.ShipAssertTag(UInt32 tag, Category category, Boolean condition, String assertMessage)  
 at Microsoft.Office.Web.Conversion.Viewing.Client.Librarian.OnReceivedStoredData(IAsyncResult ar)  
 at Microsoft.Office.Web.Common.AsyncResult`1.SetCompleted(T result, Exception e, Boolean completedSynchronously)  
 at Microsoft.Office.Web.Conversion.Viewing.Client.ViewItemStore.SetCompleted(ItemRetrievalInfo info, ItemRetrievalStatus status)  
 at Microsoft.Office.Web.Conversion.Viewing.Client.ViewItemStore.OnReceivedItemFromBackend(IAsyncResult ar)  
 at Microsoft.Office.Web.Common.AsyncResult`1.SetCompleted(T result, Exception e, Boolean completedSynchronously)  
 at Microsoft.Office.Web.Conversion.Framework.Remoting.AsyncGetItemInfo.SetCompleted(ItemCacheState cacheState)  
 at Microsoft.Office.Web.Conversion.Framework.Remoting.ConvertingServiceProxy.OnItemDataReceived(IAsyncResult ar)  
 at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)  
 at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(IAsyncResult result, Boolean completedSynchronously)  
 at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  
 at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)  
 at System.ServiceModel.Security.ApplySecurityAndSendAsyncResult`1.SharedCallback(IAsyncResult result)  
 at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  
 at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)  
 at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.OnGetResponse(IAsyncResult result)  
 at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  
 at System.Net.LazyAsyncResult.Complete(IntPtr userToken)  
 at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)  
 at System.Net.ContextAwareResult.Complete(IntPtr userToken)  
 at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)  
 at System.Net.HttpWebRequest.ProcessResponse()  
 at System.Net.HttpWebRequest.SetResponse(CoreResponseData coreResponseData)  
 at System.Net.ConnectionReturnResult.SetResponses(ConnectionReturnResult returnResult)  
 at System.Net.Connection.ReadComplete(Int32 bytesRead, WebExceptionStatus errorStatus)  
 at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)  
 at System.Net.LazyAsyncResult.Complete(IntPtr userToken)  
 at System.Net.ContextAwareResult.Complete(IntPtr userToken)  
 at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)  
 at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)  
 at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)