T500 64-bit PRO/1000 Interrupt Storms

[rvacha]

I've got an odd problem and I'm not sure what to try next. Summary: After a couple of minutes to a couple of hours my CPU activity will jump to 40-50% usage. ProcessExplorer indicates that I have an interrupt storm, but it cannot identify the culprit. Kernview doesn't work on 64-bit platforms so I have been struggling to zero in on the problem. What I can say is the following:

1. The storm will occur faster if both Ethernet AND USB are plugged in
2. The machine will never go to sleep by itself once a storm starts
3. Forcibly sleeping the machine (F4 or Vista menu) will stop the storm - for a while
4. Once the first storm has occurred subsequent storms seem to happen quicker
5. If Ethernet is left physically unconnected the storms will still occur, it just takes longer
6. If the X4500 graphics drivers are removed, no more storms
7. If Ethernet is disabled in BIOS, no more storms

I tend to think that Ethernet is the root cause, but I've tried three different driver revisions (and five graphics versions) without success.

Machine:

T500 2242-CTO
Vista 64 Business with SP2 (I had the problem before SP2 as well)
No switchable graphics
Latest BIOS 2.12 (I had the problem on earlier versions as well)
Intel SSD

I cannot rule out the possibility of a hardware problem, so I am hesitant to do a full restore. Any clues?

[GomJabbar]

Check the IRQ's that the suspect devices are using in Device Manager. Maybe you can change the IRQ(s) on one of the devices to a different one. Check Control Panel > Administrative Tools > Event Manager, and check your C:\WINDOWS\Debug\UserMode\userenv.log file for additional clues.

[jdhurst]

I saw this same complaint in Experts-Exchange. How do you know interrupt activity is causing CPU usage? Process explorer may need some interpretation. I ask this because I have not ever seen interrupt activity create CPU usage. No explanation was given in EE either.

What does Task Manager say when this is going on? That is, what processes are eating up CPU?

Might you have a virus? or other malware?

For sure, in a properly set up Windows 2000 IBM laptop, a properly set up Windows XP Pro IBM laptop, or a properly set up Vista Business IBM laptop, this has never happened to me or to any client.

I won't say it cannot happen, just that it is unique in my very broad experience and surprising.

Check also for hardware errors (PC Doctor). Set Task Manager to run and to run minimized in the System Tray. When the CPU usage goes up, it will turn bright green. Restore it and tell us what processes are running (sort by process).

Also, Ethernet drivers by know (a decade or so later) are very highly reliable and robust. But by all means, try re-installing your Ethernet drivers if you suspect them.
... JDH

[rvacha]

Thanks for the reply. I've got no userenv.log file and all IRQs are grayed out so I can't change them. There are no conflicts shown. I've checked out the event viewer several times and it doesn't provide a clue as far as I can tell. It can show interrupts, but not the source of them. No warnings related to this problem can be seen IMHO. I've installed xperfview Performance Analyzer but was not useful either. From what little I could find on the web storms frequently cause DPC errors, but I've got none of those either

Driving me crazy

[rvacha]

JD-
The standard task manager just shows random "blips" of CPU activity, no more than 10% total for all processes combined, yet its little green icon in the notification area is running 50%. I installed Process Explorer from Sysinternals and it shows "Interrupts" consuming 40-45% of the CPUs. Unfortunately it does not have the ability to pinpoint the source. Fan runs constantly, Vista sees activity that causes it not to sleep, so I know interrupts are the problem. I've tried the Lenovo Ethernet drivers as well as two different driver packages from Intel and they all suffer from this problem. I always unistall with "delete the driver software" selected and then reboot before installing drivers again. No luck.

I run NOD32 and have no malware according to it.

I've tried disabling almost everything non-essential in BIOS (Firewire, serial, parallel, modem, fingerprint, etc) and that hasn't helped - unless I disable Ethernet

[rvacha]

One other odd thing (at least to me). It appears that when the Ethernet settings are changed in device manager it resets, momentarily losing connection. This however does not help, nor does disabling Ethernet in device manager. Only forcible sleep or reboot will clear it

[jdhurst]

JD-
The standard task manager just shows random "blips" of CPU activity, no more than 10% total for all processes combined, yet its little green icon in the notification area is running 50%. I installed Process Explorer from Sysinternals and it shows "Interrupts" consuming 40-45% of the CPUs. Unfortunately it does not have the ability to pinpoint the source. <snip>

Something does not add up. If the system tray icon is half green, then CPU must be more than 10 percent. It simply must. Open Task Manager and click on the Performance Tab. The CPU window on the left should mirror the Task Manager icon in the system tray. If it does not, something is wrong.

I think System Idle Process may use an interrupt (I admit to not being certain here), and it always says it is using 98% ocf CPU, but it is just like a place holder and doesn't use real CPU. So there may be something else that we are not seeing.
... JDH

[rvacha]

I realize it sounds odd, but that's the way it is. It appears the gage sees more than what is reported under processes. The 50% task manager gage is consistent with a sidebar CPU speed gadget I have installed - but it's possible they both get their info from the same (kernel?) source. The CPU window in task manager does indeed agree with the gage - but the numbers on the processes tab don't add up to 50%. Process Explorer treats system idle process and interrupt as separate entities. When a storm hits system idle time drops to roughly 50% and interrupts jumps up to approximately 50%

[K0LO]

I had a problem like this last year on a desktop machine. To make a long story short, it occurred after installing software that uses a virtual network adapter (Cisco VPN client or Virtual PC). Unbinding Virtual Machine Network Services (for Virtual PC) or the virtual network adapter for Cisco VPN would stop the interrupt storm. My adapter is different from yours but I found an updated Broadcom driver on their web site that specifically listed this problem in the change log for the latest release. Upgrading the driver fixed the problem.

[rvacha]

Murphy's Law mandates that as soon as I post that things appear to be working, things will probably blow up, but here's what appears to be working

Advanced tab - Interrupt Moderation = Enabled
Advanced tab - Performance Options - Interrupt Moderation Rate = Medium. Adaptive and Extreme do NOT work. High appears marginal (i.e it takes a long time to invoke a storm)

Oddly enough Advanced tab - Interrupt Moderation = Disabled does not work

For a long time I thought my problems were rooted in the GMA (X4500) driver since this thing is super buggy. I ask it to pump maximum pixels (24-bit 1920x1200 VGA) and I'm guessing something (hardware, Vista, driver) is not handling the bombardment of interrupts coming at it

[jdhurst]

<snip> I'm guessing something (hardware, Vista, driver) is not handling the bombardment of interrupts coming at it

It may be a hardware problem, and the driver for your machine may be a problem (unless it is an ubiquitous Intel NIC), but I don't see it as a Vista problem (not enough complaints about this).

Are you doing something to attract a bombardment of data or something coming at your machine. The neat thing about interrupts is that they respond. If nothing comes at them, they do nothing (just sit there and wait). So what might you be doing to cause this?

I am beginning to think (since you say that your system tray Task Manager icon differs from the Performance numbers in Task Manager) that it may be worthwhile to consider rebuilding your machine.
... JDH

[rvacha]

I ended up rebuilding it as a clean install with an SP2 slipstreamed Vista 64 Business (I have a MS volume license), the latest Ethernet driver from Intel, and the second to latest X4500 driver from Intel (the latest as well as the rather outdated Lenovo driver BSOD when I plug in a projector). No dice, still have the storm. Next was to back off the BIOS from the current 3.00-1.03 to the one that shipped with the T500, 1.20-1.01. That move helped somewhat for several days but it seems to be back to its old ways. Changing the 82567LM Ethernet from Adaptive interrupt moderation rate to Medium PLUS disabling reducing link speed during idle seems to help a little.

I am at the point where I will consider the latest RC build of Windows 7, but if I do this I would do a clean install (as opposed to an upgrade) and then have to rebuild the entire machine once again

On what may be a related note the current installation has a weird problem where the "Network" folder in Windows Explorer and the "View Full Map" in the Network and Sharing Center stop working after a while, usually after resuming from sleep. Mapped network folders and internet never bust. Its as if LLTD just decides to go to sleep. One thing that bothers me is that when I open Client for Microsoft Networks and select Windows Locator as the name service provider the selection doesn't "stick" - if I open it up again it shows blank, nothing selected. If I select DE Cell Directory Service it sticks, but of course that busts my connections. I have Googled this and found others with the same problem but there's never a resolution. So I tried uninstalling the Client, rebooting and reinstalling without luck.

I am beginning to get extremely agitated. I want to shoot something and put it out of its misery but I can't figure out what to shoot first

[jdhurst]

I ended up rebuilding it as a clean install with an SP2 slipstreamed Vista 64 Business (I have a MS volume license), the latest Ethernet driver from Intel, and the second to latest X4500 driver from Intel (the latest as well as the rather outdated Lenovo driver BSOD when I plug in a projector). No dice, still have the storm. <snip>
I am beginning to get extremely agitated. I want to shoot something and put it out of its misery but I can't figure out what to shoot first

I have to go back to square 1 for a bit. Any machine I preloaded with IBM/Lenovo's recovery CD's has worked properly (Windows 2000, Windows XP Pro and Windows Vista Business). Once the newer ones settle down (updates and indexing), CPU usage is about zero.

And you are saying you build fresh and right away CPU usage is a problem.

Maybe the machine has a defect. ... JDH

[rvacha]

Yes, fresh build but not from recovery since I dislike/can't use most of the Lenovo stuff. And yes, it DID work when I first got the machine. The problem is either:

1. Somewhere between SP1 and SP2 windows update changed something for the worse. I tried uninstalling ALL of the OS-related updates in batches (by date installed if possible) all the way back to SP1 but this didn't fix anything
2. Marginally good hardware has become marginally bad hardware

I've run everything I can think of including PC Doctor, Intel's Proset diagnostics, DX10 diagnostics, etc and nothing fails. Ran memtest overnight with no fails. I've run these (except for memtest obviously) before and during a storm with no fails - nothing seems to think there is anything wrong with interrupt storms. In general I'd say the biggest problem I've had is a lack of good 64 bit tools so I'm not so sure these diagnostics can even "see" a problem. Vista reliability and performance tools will plot interrupt activity but can't pinpoint the source. Several activities ramp up a tad bit (usb.sys, hecix64.sys, WDM.exe, DPC, iastor.sys from memory) but again it is impossible for me to pinpoint the root and I can say that the number one most active area are interrupts - at least according to Process Monitor and Vista

I don't think I'll get Lenovo to swap the machine (or parts) if I can't show a hard failure and reverting back to the recovery image is not a "fix" in my book since I would never be able to upgrade security issues, drivers, etc. In fact the factory install X4500 driver BSODs every time I plug in a projector and is unusable to me. One time it crashed so badly that Vista thought it was no longer a notebook PC and I had to recover the stupid machine to get ACPI back. Then there's the possibility that if I do another recovery I'll find I still have a problem due to hardware.


I might still contemplate installing Win 7 to see what happens but a plethora of solid drivers does not appear to be its strong suit right now

[jdhurst]

From your last post, you may have created your own problem.

No one has ever succeeded in convincing me that "clean builds" are worth the time of day. It is faster (much faster) and easier to preload and remove the 3 to 6 things one doesn't need and move on, than it is to use some other CD to install and hunt down all the necessary drivers. Given the plethora of chipset and other hardware drivers, the results usually will not be as good as a preload unless you spend an inordinate amount of time correcting things.

I have had to do clean installs on non-IBM gear. The time to get it correct was excessive (which my clients must pay for) and the results are not as good.

If you call IBM support (what I do if I have a problem that looks and smells like a hardware or base OS problem) they will insist on a preloaded environment before servicing.

Each person will have their own reasons for doing things, and I offer only my working (and successful) experience here. But when you tell me you have the same problem that (heretofore) I have not heard of, and then, by the way, it started up after you set up your clean build, then therein is likely the root of the problem.

... JDH

[rvacha]

I never said the problems started after a clean install. These problem have been going on for months and occurred on the Lenovo install which had Vista SP1. Yes, it did work with the Lenovo install - for about two months - and then started having storms. Knowing that the machine used to work I started removing things that windows update had installed going all the way back to square one (the original Lenovo configuration plus my apps plus a newer X4500 driver because the original Lenovo one crashes) Still didn't fix it. That is when I decided to do a clean install of Vista SP2 in hopes that a baseline install of Vista with all updates might fix things - but it didn't. Finding drivers isn't so bad since Lenovo has them all in C:\Drivers

[jdhurst]

I never said the problems started after a clean install. <snip>

My apologies. I was reading your post of May 18th.

I have not ever experienced what you have, either on my own machines, or on client machines. I haven't seen too many people post in this thread either (just a couple of folk).

I would find crashing and slowness to be distracting to the extreme and would like to solve it if it were my machine.

So I think you need to try to work out whether it is hardware (in which case Lenovo needs to fix it), or your software (you had a preload successfully running for two months). How to do this? My recommendation would be to do the following:

1. Do a complete PreLoad restore.
2. Uninstall things you do not need (Norton AV if you do something else, Message Manager, stuff like that)
3. Leave installed (at least for now) all the Lenovo drivers.
4. Update all the Lenovo drivers from the Lenovo support site.
5. Download SP1 for Vista and install manually.
6. Do all other Vista updates.
7. Do not install SP2 - it is still Beta and not in production (at least as of this past weekend)
8. Install only known, good quality software that is needed in daily work (Office, AntiVirus, Adobe, etc.)
9. Do not install games or personal software.
10. Turn of all automatic updates and startup scans (set things to download and advise)
11. Per K0lo's remark, try not to install VPN software for the first bit.

Run it like that for a while, and see what happens. If CPU usage goes up inexplicably and stays up, then contact Lenovo support.

.... JDH

[rvacha]

Thanks for the help jd - I might just end up doing that but it'll have to wait a week.

BTW SP2 is official, as both an upgrade ISO package and an ISO package of Vista with SP2 slipstreamed from MS Volume Licensing. I used the later for the clean install. The upgrade ISO package actually contains three upgrades on one disk - Vista 32, Vista 64, and Server 2008. So far I am not impressed - it breaks NOD32 antivirus (this is a well documented problem with no solution yet), the 64-bit version of IE8 randomly crashes (which oddly causes a fault in ole32.dll - problem being researched by MS), windows explorer randomly crashes, and as mentioned in a previous post network mapping of any sort will sometimes break.

As a data and CAD intense tech corporation we dumped VPN two years ago when it became obvious that 64 bit support was not going to happen (we've migrated to SSL) so I've never had VPN installed.

[jdhurst]

OK, thanks for the feedback. Two points:
1. I still do not see a production release for SP2 and ZDNet said the same thing today. If you have a link for a production (not beta) SP2 file, please let us know.
2. VPN for Vista 64-bit has been a thorn. I worked with SafeNet-Inc after licensing their 64-bit version of IKE VPN, and now it does work. I use it all the time and reliably. It seems to want to leave lots of errors in the event log, but I think that is just because it normally does not find itself connected. Also, it conflicts with active tunnels in my LinkSys RV042 at home. Multiple tunnels to the same site confuse things. But these errors are bland, never show up unless I look, and do not effect reliability.

... JDH

[rvacha]

SP2 IS available, but not through windows update yet. You can get it through Technet, MSDN, or in my case Volume Licensing Services. To get it one of these ways you have to have a license (volume licensing) or be a subscriber (TechNet and MSDN)

http://community.winsupersite.com/blogs ... -isos.aspx

Plain old SP2 upgrade is also available through these channels

[jdhurst]

Thanks. What you are posting, however, are still beta SP2 releases. Microsoft has been coy about giving a date for Production SP2 release, apparently because Vista has failed (even though we use it and it can be made to work), and they want the hoopla for Windows 7. Everything posted at this point must be removed when the production release comes out (according to Microsoft). ... JDH

[rvacha]

No, it's the real final. Period.

[jdhurst]

No, it's the real final. Period.

Well perhaps where you are. Everywhere I have look and read - everywhere - SP2 is not released.

Now, for ALL services packs from Windows 2000 forward, the day released is the day you can download from Microsoft and I still cannot download SP2 from Microsoft. So I will get it when Microsoft releases it as they are truly the final authority.

Thank you, though for the links and such. ... JDH

[jdhurst]

SP2 Final came out today. I don't know if this file is bit-wise the same as the earlier "final", but I was happy to wait until today. No problems of any kind (at least after review of critical items). ... JDH

[rvacha]

As an FYI I installed Win7 x64 RC (build 7100) and then later build 7232 and the interrupt storms happen less frequently.