Weird problem with T520, now won't start

[lhp]

Thursday I had an unannounced power outage due to digging. My T520 ran fine on battery, but as I didn't know how long it would take for power to return, I shut it down as a precaution. Normally I have it running nearly 24/7 without problems, since I bought it as a refurbished machine in late 2017.

When I restarted the machine, it came up fine, but after 30 minutes it turned off without warning. I investigated, thinking it was overheating, but no, temperature was okay, and it consistently died after 30 minutes. A search suggested this was a problem with the Intel Management Engine. I upgraded the BIOS from 1.39 to 1.49 with an ISO from Lenovo, thinking this might help. It still started fine, only to die after 30 minutes, whether I booted an OS (I even tried to boot the Windows10 partition it came with, which I had never used before, as my normal OS is Xubuntu), launched Memtest86 or left it on the BIOS config screen.
Further searches suggested resetting the machine by removing the motherboard battery in addition to power and main battery, and leave it for some time. It was at that point, the machine no longer booted. I tried several paths of troubleshooting, leaving the MB battery out for 15+ hours, removing RAM, etc. All to no avail.

I have tried a "professional" repairman, who just phoned me today saying they had it up once, but only that one time. He has no diagnosis and no fix. (Fortunately also no charge.)

It seems to me that this must be something utterly trivial that keeps the machine from booting, but I am not a hardware expert (although I can do a little), and don't know what it could be. I haven't been able to dig up very much on this bug, other than already mentioned.

Anybody here who can help with advice? What is broken, and what should I replace? I can't afford a new machine at the moment, and though I have an older spare laptop, I do miss the 16 GB RAM I had in the T520. (Specific model number 4242W38, if that is of any use.) Where does the Intel Management Engine store its config, and how do I reset it? Would changing just the CPU (if possible?) make any difference, or do I need to replace the motherboard? What other options do I have?

[RealBlackStuff]

Disable it in BIOS and uninstall/delete any Intel ME program(s).

[atagunov]

So you can still get into BIOS but not boot?..

Obviously RealBlackStuff got lots more experience so please do as he says first, I just wanted to note..

..are you a sufficiently hands-on person when it comes to software a bit like myself? For somebody similar to myself I'd suggest trying to flash CoreBoot before changing the CPU. If you already have a Raspberry Pi then the only other hardware you would need are probably hooks like these ones. Disclaimer: although I own these hooks I haven't tested them yet. If you don't have a Raspberry Pi you'd need a USB programmer and one more computer running under Linux I think. 2nd disclaimer: I haven't flashed CoreBoot either yet CoreBoot is an even better way to get rid of ME I think if you configure if correctly. Maybe for somebody wielding a Raspberry Pi or USB programmer there are other ways to disable Intel ME, other than flashing CoreBoot.

BTW do we think it might be EC (embedded controller) misbehaving? I'm not really sure where EC firmware resides.. I know it is updated as part of installing a new BIOS (you need to boot machine for that..) but I'm not really sure where this firmware sits.

That said I personally have no clue what might be wrong with the machine. Replacing the CPU seems like a somewhat random suggestion. However for all I know any part can be broken and replacing the CPU is as much a shot in the darkness as replacing anything else. The other obvious candidate for replacement is the motherboard if we want to go this route. I'd say that may probably have a better chance of curing the issue..

[kfzhu1229]

Thursday I had an unannounced power outage due to digging. My T520 ran fine on battery, but as I didn't know how long it would take for power to return, I shut it down as a precaution. Normally I have it running nearly 24/7 without problems, since I bought it as a refurbished machine in late 2017.

When I restarted the machine, it came up fine, but after 30 minutes it turned off without warning. I investigated, thinking it was overheating, but no, temperature was okay, and it consistently died after 30 minutes. A search suggested this was a problem with the Intel Management Engine. I upgraded the BIOS from 1.39 to 1.49 with an ISO from Lenovo, thinking this might help. It still started fine, only to die after 30 minutes, whether I booted an OS (I even tried to boot the Windows10 partition it came with, which I had never used before, as my normal OS is Xubuntu), launched Memtest86 or left it on the BIOS config screen.
Further searches suggested resetting the machine by removing the motherboard battery in addition to power and main battery, and leave it for some time. It was at that point, the machine no longer booted. I tried several paths of troubleshooting, leaving the MB battery out for 15+ hours, removing RAM, etc. All to no avail.

I have tried a "professional" repairman, who just phoned me today saying they had it up once, but only that one time. He has no diagnosis and no fix. (Fortunately also no charge.)

It seems to me that this must be something utterly trivial that keeps the machine from booting, but I am not a hardware expert (although I can do a little), and don't know what it could be. I haven't been able to dig up very much on this bug, other than already mentioned.

Anybody here who can help with advice? What is broken, and what should I replace? I can't afford a new machine at the moment, and though I have an older spare laptop, I do miss the 16 GB RAM I had in the T520. (Specific model number 4242W38, if that is of any use.) Where does the Intel Management Engine store its config, and how do I reset it? Would changing just the CPU (if possible?) make any difference, or do I need to replace the motherboard? What other options do I have?

Seeing that you had it running 24/7 and with this sort of symptoms it seems like the capacitors on your motherboard are dying. Had this symptom on my ThinkPad 770ED before it died. Also had this on a Latitude D830 motherboard before it wouldn't even charge the battery and that turned out to be a dead high-current resistor, replaced that with a regular resistor with 4 rings and that worked fine ever since.
To be clear, what does it mean by the machine no longer booted? Shuts off before Windows completes boot? During/after ThinkPad splash screen? No splash screen but the power LED comes on? Power LED doesn't come on but the battery does charge? Or just totally dead where not a blink of the battery LED can be seen?

[lhp]

Sorry if I was unclear. At first I could boot the machine into Linux, Windows, Memtest86 and BIOS settings, but it would shut off hard after 30 minutes.

After removing the CR2032 MB battery (and reconnecting it after some time, first I tried 15 minutes, then 15 hours), the machine no longer starts. Nothing is displayed on the screen. The battery status light blinks normally (I think: two short blinks, a pause, repeat) when battery and power are connected. The light in the power button lights up, and also the keyboard lamp sometimes blinks, when I press the power button, but other than that, nothing happens, screen stays black, no activity from the disk. I have tested the CR2032 battery, and it shows 3.32 V, so should be okay, and as far as I can tell, it is not possible to connect the wrong way round.

[lhp]

Seeing that you had it running 24/7 and with this sort of symptoms it seems like the capacitors on your motherboard are dying.

That sounds plausible. Before this happened, I have not had any problems having the machine turned off for say a day or two (although this rarely happened), but in that case, there would still be some power from the MB battery keeping it in a startable state, I suppose?

I think I answered your questions in my previous post, sorry for the unclarity.

Would there be any simple way to pinpoint which capacitor or capacitors are failing?

[kfzhu1229]

Sorry if I was unclear. At first I could boot the machine into Linux, Windows, Memtest86 and BIOS settings, but it would shut off hard after 30 minutes.

After removing the CR2032 MB battery (and reconnecting it after some time, first I tried 15 minutes, then 15 hours), the machine no longer starts. Nothing is displayed on the screen. The battery status light blinks normally (I think: two short blinks, a pause, repeat) when battery and power are connected. The light in the power button lights up, and also the keyboard lamp sometimes blinks, when I press the power button, but other than that, nothing happens, screen stays black, no activity from the disk. I have tested the CR2032 battery, and it shows 3.32 V, so should be okay, and as far as I can tell, it is not possible to connect the wrong way round.

After your first boot after reseating the CMOS battery, you tried waiting for 10-15 minutes and see if the screen would come up right? My T41 did that same behaviour after BIOS update and CMOS battery unplug and it took 5 minutes for the screen to come back up
Then I think the only thing left to do is to strip the system down to bare essentials including power, CPU, RAM, LCD, keyboard, cmos battery, reseat all of these and see if anything changes then.
I think unfortunately you are likely to be looking at failed capacitors and such and they can be an absolute pain to deal with short of tools such as a bench power supply.
I was just lucky with my Latitude D830 because I was just randomly touching the PCB and then it started turning on, and shuts right off as soon as I release my finger, and that the resistance value of the resistor matches exactly with the resistance of my thumb, and that Latitudes actually have ALL of their components labelled on the motherboard.
Does your board have the NVS 4200M graphics? If so it could also be dead graphics (although these laptops don't have any design flaws around the graphics chip so the chance of that isn't all that high).
Don't think it is a failed BIOS update though as it did fire up just like before with the BIOS update.
I might have a spare T520 board real soon though.

[lhp]

After your first boot after reseating the CMOS battery, you tried waiting for 10-15 minutes and see if the screen would come up right? My T41 did that same behaviour after BIOS update and CMOS battery unplug and it took 5 minutes for the screen to come back up

Hmm. I don't think i ever waited that long. You mean you pressed the power key briefly, then waited five minutes for the machine to power up?

I think unfortunately you are likely to be looking at failed capacitors and such and they can be an absolute pain to deal with short of tools such as a bench power supply.

I was so lucky to get in touch with a guy who is so "crazy", he does bug finding and troubleshooting just for the fun of it! As a hobby! I will be shipping the machine to him next week.

Does your board have the NVS 4200M graphics? If so it could also be dead graphics (although these laptops don't have any design flaws around the graphics chip so the chance of that isn't all that high).
Don't think it is a failed BIOS update though as it did fire up just like before with the BIOS update.
I might have a spare T520 board real soon though.

No Nvidia graphics IIRC. But it has 1600x900 display, and I had it upgraded to 16 GB RAM, expecting it to serve me for at least one or two more years. Model 4242W38, I can't seem to find a page with the exact specs of that model.

[MikalE]

Sometimes, but not always, electrolytic capacitors that have failed will be swelled or even leaking in some cases. For SMD's though, you will nearly always need test equipment to find the buggered component unless you see a hot spot or burned mark on the MB.

[lhp]

Good news everybody! (Sorry for the bad Farnsworth impersonation.)

I fetched the machine from a repair shop that had given up on it, the day before yesterday. Since, I have - just for the heck of it - tried pushing the power button on it now and then.

Today, I attached the power cable, and tried once more. It lit up the power button, blinked the keyboard downlight, as usual in the first attempt. As I had seen before. The second time, though, it rattled a bit, as if the DVD-drive got power or was searched for a boot CD, and the speaker mute button also lit up. On the third attempt, the machine took a few seconds, then continued to come up, displayed something about some diagnostic (had I known, I would have filmed it - these messages flicker over the screen so fast, they are nearly useless otherwise), I pressed F1, and found myself looking at the BIOS configuration screen. It has been sitting with that screen on, in the Date/Time tab, now displaying 01/01/2001 00:27:00! Three minutes to potential ME shutdown... I am excited!

[lhp]

...aaaand then it shut down right on the second at 00:29:59.

Fortunately it came right up fine when I then pressed the power button. It is now at 00:27:14, and I will be recording the shutdown and hopefully the subsequent powerup in a few minutes.

[RealBlackStuff]

Like I said before: disable ME.
Unfortunately not in BIOS, but read this: https://wiki.gentoo.org/wiki/Sakaki%27s ... ent_Engine

[lhp]

Like I said before: disable ME.
Unfortunately not in BIOS, but read this: https://wiki.gentoo.org/wiki/Sakaki%27s ... ent_Engine

Well, yes, perhaps. I just wonder if there is a less "severe" solution. For instance, I haven̈́t tried booting in Windows10 and run the Lenovo iME firmware upgrade. Actually; I still haven't found any hard evidence (except for the obvious, that it _is_ there on the CPU architecture the machine has) that the IME is present. The one time I booted it in Windows 10 (delivered on the machine from the refurbishing reseller) I found no trace of drivers or anything related to IME; but as I believe I mentioned, I am not at all well-versed in Windows intrinsics.

[lhp]

(Goodness, how I just hate debugging hardware/firmware issues.)

Now I have downloaded https://download.lenovo.com/pccbbs/thin ... _linux.iso and made a bootable DVD. Hoping diagnostics might shed some more light on what is wrong with my machine (or find other problems, perhaps.)

It boots fine, but just ends up with a gray background and an arrow mouse cursor, nothing else happens. (Well, after 30 minutes, the machine powers off, but that is to be expected.)

I typed in my serial number, so this image should be useful for my machine. The documentation PDF is not helpful about booting the diagnostics image or launching the diagnostics program. What is going wrong?

[lhp]

Will I never get a break?

Found this page https://forum.mxlinux.org/viewtopic.php?f=108&t=57126 about the Lenovo Diagnostics (Linux) iso. TL;DR: It is broken. Why does Lenovo put up a broken system on their support page? It is the image they recommend for all Thinkpads, they clearly stated on the page, fairly recently: "Note: Lenovo Solution Center discontinued support after 2019-04. Instead, use the "Lenovo Diagnostics" Windows version or the "Lenovo Diagnostics" Linux Bootable version to diagnose your PC." Yet that same Linux Bootable version is trash.

[lhp]

Finally, a break of sorts.
Got all the RAM installed again. Tried looking at ME software for windows - which didn't work with Windows10Home for some reason. Shut down Windows, which then insisted on running some updates, I suppose that's how Windows works nowadays, I don't use it. Have no idea what this may have done to the machine. Anyway. Rebooted, have found that sometimes it won't boot with the battery installed, although it charges fine, and runs fine from it when the power cable is disconnected. Looked at the BIOS settings again. Tried to switch boot from Legacy/UEFI to UEFI/Legacy - no idea what difference that should make. Found a page talking about entering the ME interface by pressing the blue ThinkVantage button, then control-P. Tried that. Got an error message ME is in Recovery State, and a page of hex data, ending with:
"MEBx Continue State: 0001
MEBx Error State: 0303"
Repeated it, to capture the message with my phone camera. Once I got "ME FW Update Failed. Please try again!" Went to research these messages. Booted the machine up in MemTest86+ in the meantime.

And lo! To my great surprise, the time passed the 0:29:45 point unnoticed, and was at 0:45:something when I glanced at the machine! The MemTest had completed without errors. Left it for another half hour, then tried the same again. It has just passed 48 minutes now, no problems. Now I have to see the same happen after I boot Linux. And see how it deals with things like sleep.

This is not a solution, I'd love an explanation of what is really wrong with the machine, but it seems to be ready to work again now, and that's most important to me.

A big THANK YOU to everyone who has responded, every slight clue you provided has helped somehow. As suggested I might want to install coreboot, but I guess I will leave that until the next time this machine f?x up. I hope this thread may provide others who get to face the same or similar issues with a few clues.

[lhp]

Oh well...

The machine didn't like going to sleep. Now I don't get the ME is in Recovery State message, and of course this means the machine powers off after 30 minutes again. I can't seem to retrace what I happened to do to make it appear in the first place.

I do think however, that I know how I "broke" it again: in Linux (XUbuntu 18.04) I clicked Suspend, and when I pressed thte power button to wake it up, it stayed in sleep. The subsequent removal of power to restart the machine resulted in the reappearance of the error. I did notice in dmesg that the mei kernel modules had loaded when running with the ME in Recovery State, though.

It is almost as if there are two sets of firmware in there: sometimes I get the large lenovo logo, sometimes I just get the BIOS name etc. Sometimes it shows the prompt to press ctrl-S for PXE boot options, sometimes it doesn't. By now it seems as if the battery may be involved somehow. I am absolutely stunned.

Oh my, I yearn back to simpler times, when there was a power button and a reset button that connected directly to the motherboard, and if hardware ddn't work, it was because it was physically broken, not because it was in some weird limbo configuration state.

[lhp]

Noting that the interest is not that great, judging by comments, I guess this will be me last comment on this matter.
I must ashamedly admit, that I did not attempt to go forth in any scientific manner, logging everything I did. However, by persistent rebooting (ctrl-Alt-Del from the GRUB menu), powering off the machine with the power button or pulling the power cable and battery if that didn't work, and powering it up without the battery, inserting it at various times, and trying to enter ME by pressing ctrl-P (no idea if that had any significance), I managed to get the ME to start up in the recover state again, which seems to work a lot better than the other state. This time, the MEBx Error State was 0304 rather than 0303, but the machine still runs fine without powering down "unexpectedly" (well, kinda expected after 30 minutes in the "other" state.) Of interest is that the dmesg from Xubuntu now lists two me kernel modules, just as it did the previous time I had the machine up with the ME in this state. As I usually keep the machine running 24/7 without sleep I guess I can live with how it works now, at least for the forseeable time, even if it feels like it is mostly running by my willpower and stubbornness more than anything.

[lhp]

So now the machine has been working perfectly, even surviving reboots and shutdowns for three months. Today I did some system maintenance and did a shutdown instead of just rebooting. Lo and behold, the ME timeout problem has reappeared. I post here for continuity of the story, and to ask if anyone might have anything new to add regarding IME loading problems on the ThinkPad T520. I have tried downloading coreboot and me_cleaner, but the utility intelmetool to dump information states that the hardware version (SandyBridge?) is not supported by it. If I weren't a mostly rational person, I'd say my machine was cursed...

Observation: For a while, a status screen showed up right after asking to enter PXE Boot options with Ctrl-S, I could then press Ctrl-P (I think) to enter the ME configuration, not that I could do anything much there, but at least it seemed to load alright. Now, I don't get that of course. But the PXE boot prompt does not always appear when i boot or reboot the machine. It seems it has to boot from a power-detached state to show that prompt. Is there any logic to this?

Thanks in advance for any input anyone might be able to share.

Oh well, meanwhile I'll have to try the "chickenwaving" repeating reboot tricks I had some success with three months ago...

[atagunov]

Get a hardware programmer or Raspberi PI/clip and mess with BIOS the hard way?.. I never tried me cleaner myself but I suspect it should work best with a hardware programmer

[RealBlackStuff]

If you run that POS W10, PXE will only show up after a complete shutdown, which normally stupid W10 prevents...

[kfzhu1229]

Sorry for reviving this near 1 year old thread, but it seems like I have the Management engine stuff to deal with now on a Dell laptop.
If someone here have successfully gotten around the 30 minute shutdown problem, can you tell me that if the fix mentioned by RBS is permanent? Or does the fix only work for a random amount of time before the 30 minute shutdown is back again?
Also, from what I see online, if I install Management engine properly the shut down won't occur. However, does the software for these Sandy bridge and Ivy bridge machines work properly in Windows 10? Every time I try to install the software I get the hardware is not supported error, yet the 30 minute shutdown is definitely there and also I have two BIOS chips mounted on an extension board.
Does ThinkPads with the ME enabled out of the factory also have two BIOS chips like this? I noticed one chip is more empty than the other chip, so do you think the emptier chip is the one of interest?

[kfzhu1229]

Actually, turns out dumping the BIOS from another machine that came with ME disabled out of the factory did the trick!
If your ThinkPad BIOS chips are easily accessible and you have a SOP8 capable USB programmer, you could just find a BIOS dump (BIOS + EC both required!) and then that could also get around the 30 minute shutdown problem for good!
Also I believe the me_cleaner utility doesn't work with the vulnerability patched BIOS versions.