Windows Server 2003 crashes and restarts

[jITguy]

Our three virtual servers keep crashing randomly at least once a day. Event viewer reports the following error:

Event Type: Error
Event Source: System Error
Event Category: (102)
Event ID: 1003
Date: 12/2/2010
Time: 8:44:37 AM
User: N/A
Computer: XEN06
Description:
Error code 00000050, parameter1 9599fa84, parameter2 00000000, parameter3 f5f6a616, parameter4 00000000.

For more information, see Help and Support Center at....
Data:
0000: 53 79 73 74 65 6d 20 45 System E
0008: 72 72 6f 72 20 20 45 72 rror Er
0010: 72 6f 72 20 63 6f 64 65 ror code
0018: 20 30 30 30 30 30 30 35 0000005
0020: 30 20 20 50 61 72 61 6d 0 Param
0028: 65 74 65 72 73 20 39 35 eters 95
0030: 39 39 66 61 38 34 2c 20 99fa84,
0038: 30 30 30 30 30 30 30 30 00000000
0040: 2c 20 66 35 66 36 61 36 , f5f6a6
0048: 31 36 2c 20 30 30 30 30 16, 0000
0050: 30 30 30 30 0000


I have read many posts about Error code 00000050, and most posts say something about a kernel driver issue. I've updated all drivers and softwares on the servers and still receive the crashes/errors. Anybody have any insight on this or experience this problem?

If anyone wants to see the MEMORY.dmp file, let me know as I cannot upload it as it's around 250MB. I can setup something on FTP

 

[Route44]

There is a Zip file option in the Manage Attachments button that you can use to compress that file.

 

[jITguy]

Zipped file

I tried that option. I will delete the current MEMORY.dmp file and let it happend again and upload the smaller dump file. It will happen again im assured. Once it does happen ill upload it.

 

[jITguy]

even after getting a new dump file it is 200+ MB. Any suggestions?

 

[Route44]

See if the following helps you to create smaller minidump files:

1. Right click My Computer > click Properties > click Advance tab

2. The Advance Screen at the bottom right has Error Reporting. Click this on.

3. Click Enabled error reporting and also check both boxes underneath: Windows operating system and Programs.

4. Now click Choose Programs and click All Programs. Then click Okay and that screen will close. At the next screen click Okay and that will close.

5. You are back to the Advance screen. Find the Start up and Recovery which has underneath it System startup, system failure, and debugging information. Click the Settings box.

6. Another screen will open. You want the System failure section. Enable all three boxes.

7. Under these boxes is Write debugging information. Choose the option Small memory dump (64kb). Click OK and that screen will close.

8. Click OK one more time and you'll be finished. Now the ability for your system to write minidumps should be enabled.

 

[jITguy]

I will change those settings on all my virtual servers. Is there a way to force a memory dump without restarting the servers?

 

[jITguy]

I received another crash error today and it created a mini dump. I have several that I will zip up and send. Let me know what you find! Thanks!

 

[Route44]

There is a conflict between the Citrix driver CtxAltStr.sys and the NOD32 driver eamon.sys with the latter preventing the former from loading.

I run NOD32 myself and NOD32 is one of the best of the best out there. One resolve might be to update both softwares.

Or you could go to ESET's official forums at www.wilderssecurity.com and ask for help. The community is very active and very helpful.

* Please keep us updated.

* By the way, how did you resolve the need to reboot in order to do the minidump settings?

 

[jITguy]

When I changed the settings from the full Memory Dump to the minidump it did not ask me to restart the server, if that's what you are referring to. I got the new minidup files after the server BSOD again then pulled the minidumps that I sent.

I will keep you updated on the progress I am making.

Thanks for your research!

 

[jITguy]

I researched the problem with the CtxAltStr.sys driver and it has to do with the Virtual Memory Optimization Tool. Many posts on the Citrix Forum are suggesting disabling that function.

Here's my plan of action:

Clone one of the servers having the problem with the BSOD
Make the changes on the cloned server, and restart it
Add a handful of users to that server and see what happens from there.

I'll continue to monitor that clone for about a week and if it doesn't restart or BSOD, ill make the changes on the rest of the virtual servers.

 

[Route44]

Sounds like a good plan. If it works for the next week please let us know. This will be good information for others. Thanks.

 

[jITguy]

I appreciate all the help you have been. i will update this thread again when I have finished testing.

 

[jITguy]

here's a bugcheck analysis of the server I cloned, not the one I am testing, but the original:

*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 10000050, {9e636a7d, 0, 8096a3aa, 0}


Could not read faulting driver name
Probably caused by : ntkrpamp.exe ( nt!SepSidInToken+24 )

Followup: MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: 9e636a7d, memory referenced.
Arg2: 00000000, value 0 = read operation, 1 = write operation.
Arg3: 8096a3aa, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 00000000, (reserved)

Debugging Details:
------------------


Could not read faulting driver name

READ_ADDRESS: 9e636a7d

FAULTING_IP:
nt!SepSidInToken+24
8096a3aa 0fb64301 movzx eax,byte ptr [ebx+1]

MM_INTERNAL_CODE: 0

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0x50

PROCESS_NAME: g2mcomm.exe

CURRENT_IRQL: 1

LAST_CONTROL_TRANSFER: from 8096af75 to 8096a3aa

STACK_TEXT:
f38769f8 8096af75 e4083398 00000000 9e636a7c nt!SepSidInToken+0x24
f3876a1c 8096b769 e4083398 e3757018 00000001 nt!SepTokenIsOwner+0x4b
f3876a3c 80938797 e3757018 f3876c04 00000001 nt!SeAccessCheck+0xc5
f3876a88 80935307 e4099ab0 f3876be8 00000001 nt!ObCheckObjectAccess+0x77
f3876b4c 80935932 00000001 8d3ec338 e4099ab0 nt!ObpIncrementHandleCount+0x361
f3876bac 80933ed2 00000001 e4099ab0 e4051478 nt!ObpCreateHandle+0x17e
f3876c80 8096854b e4099ab0 00000000 00000000 nt!ObOpenObjectByPointer+0xa8
f3876d30 8096861e fffffffe 00020008 00000001 nt!NtOpenThreadTokenEx+0x18d
f3876d4c 808897cc fffffffe 00020008 00000001 nt!NtOpenThreadToken+0x18
f3876d4c 7c82860c fffffffe 00020008 00000001 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
01c9fce4 00000000 00000000 00000000 00000000 0x7c82860c


STACK_COMMAND: kb

FOLLOWUP_IP:
nt!SepSidInToken+24
8096a3aa 0fb64301 movzx eax,byte ptr [ebx+1]

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: nt!SepSidInToken+24

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrpamp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 4b7a90ad

FAILURE_BUCKET_ID: 0x50_nt!SepSidInToken+24

BUCKET_ID: 0x50_nt!SepSidInToken+24

Followup: MachineOwner
---------

I see where it mentions ntkrpamp.exe and g2mcomm.exe. g2mcomm.exe is referring to the gotomeeting software we use for training, but not sure what ntkrpamp.exe is referencing. Any ideas on this?

 

[Route44]

ntkrpamp.exe is a core Windows OS driver. The thing with OS drivers is that they are usually too general to be of much diagnostic help. They more or less say there is a problem rather then pointing to a specific cause.

 

[jITguy]

Epilogue

I appreciate all the help you have been. i will update this thread again when I have finished testing.

I finished testing with great success!! I did some more research on the problem between ESET and the Citrix driver issue. I found that updating my version of ESET NOD32 to the current version and disabling the Citrix Virtual Memory Optimizing tool did the trick.

Here is a link to disable the Citrix VM Optimization tool: http://support.citrix.com/article/CTX117669

I haven't experience any further BSODs or server hangs since. As always, I will continue to monitor server performance for any discrepancies.

Again, thank you for all your help and I hope this thread provides useful information to server admins in the future!

 

[Route44]

Excellent! Thanks for the update and the resolve. You are correct, it will be useful to not only here but to people who will be researching for possibly the need for this solution.

What version of NOD32 did you update from and did you go with version 4? I run 4and love it.

 

[jITguy]

Epilogue

I updated from Version 4.0.467.0 to the latest version 4.2.67.10. Thanks for all your help!