Jump to content

An odd Sherman mainboard failure: help me understand what happened


supercurio

Recommended Posts

@supercurio, I'm really sorry to hear of your misfortune but glad to hear that it did not cause you to fall and get injured. However, from your description of the hill testing you were attempting at your relatively low weight, I would not expect the Sherman to fail unless the board was actually faulty to begin with.

As you so rightly mentioned, the Sherman is not known for low speed high stress hill climbing, but that does not sound like the stress you were subjecting the wheel to whilst setting up your side pad positions and you did attempt this with due caution.

You mentioned that there was some condensation/water droplets but could not think how it would have got in there? Is it possible that on a previous ride you got the wheel wet in the rain/puddles and possibly some moisture had not dried out before your pad set up session today?

The board version you have fitted is the same L20-42 version that I have had sent to me on 2 occasions as replacements after my "ECHAr" charger fail issues, but have both been solid in performance away from the "ECHAr" issue! Still, that does not rule out that you may have been unlucky and ended up with a weak/compromised board with less than perfect components or solder joints!

The most current motherboard version though, appears to be L21-10 I believe (which supersedes the L20-42) as I have mentioned in recent posts elsewhere, but I'm unaware of the exact changes the latest version motherboard offers.

4 hours ago, supercurio said:

Balancing, charging, the display, wireless data and power delivery still work, the wheel still rides.
But it's hard to know if it's safe to, even if I if hacking together power for the fans with a buck converter. Without fans, it overheats quickly with normal riding.

Now let's see what Leaperkim says :(

I would Not even attempt to ride it in it's current state and certainly would Not try any hacks to get the fans working or repair the failed component, as there is most likely more microscopic damage to the tracks within the board.

An 11 day old Sherman that failed whilst braking most definitely needs a New motherboard as the Only option to repair the wheel, but at least you will get the very latest L21-10 version as a replacement.

I wish you luck, but Leaperkim should see you right without issue may just take a little while is all.

Edited by fbhb
  • Like 1
Link to comment
Share on other sites

7 hours ago, supercurio said:

A few minutes later, the EUC World 60°C℃ alarm was triggered so I stopped and started riding on the flat to get some airflow to the board.
Later I noticed that the board temperature didn't really go down as usual, I thought I was just riding harder than usual due to the pads.

A couple km later, I stopped and smelled an odour of burned electronics around the mainboard box, and was surprised that 60-64°C was enough to give off a smell since the board was clearly not blown.

Temperature is reported from some IC on the motherboard. Like the gyro chip. So it is just some measurement of the average compartment air temperature. And the self-heating of this IC.

So for short and high burdens one will not see any real temperature rise but already has fried components on the board.

Although in your case the motor driver mosfets seem not to be overburdened and fried, they are best saved by a phase/motor current alarm. There is no way they can stand some 160-220 A beside very short and seldom peaks.

Same for the battery current - more then 70A are enough to desolder the battery wires from the mainboard ... So here only short and seldoms peaks are possible without damages...

The design and the burdens are just borderline. 

  • Like 1
Link to comment
Share on other sites

7 hours ago, fbhb said:

@supercurio, I'm really sorry to hear of your misfortune but glad to hear that it did not cause you to fall and get injured. However, from your description of the hill testing you were attempting at your relatively low weight, I would not expect the Sherman to fail unless the board was actually faulty to begin with.

As you so rightly mentioned, the Sherman is not known for low speed high stress hill climbing, but that does not sound like the stress you were subjecting the wheel to whilst setting up your side pad positions and you did attempt this with due caution.

Yes I made sure to avoid the low speed hill climbing known issue, but I wonder if by riding the hilly street faster - and drawing more amps I could have created the same situation?
Note: estimated battery current by EUC World

Part 1: https://goo.gl/maps/bk9B6hNUhDE6aK9n8 - 17% grade

1623281831_Screenshot2021-07-08at12_42_10.png.4c004e3f977d55793f1885bd9a533a3d.png

Part 2: https://goo.gl/maps/RQMU5Sk2UGwawWCz7 - 15% grade

 426715572_Screenshot2021-07-08at12_43_21.thumb.png.75b9162571130ad3b728552304493d28.png

Or did the failure happen when hard breaking?

448154263_Screenshot2021-07-08at12_45_23.thumb.png.c27e17a46bc4ff9e24fa0786cac0cd44.png

 

Of course, I'd like to understand in order to avoid another failure.

When I see riders like Chooch, Adam from Wrong Way or the recent races when they hang off the pads during acceleration and braking, my 2 accelerations don't seem that crazy.

 

Quote

You mentioned that there was some condensation/water droplets but could not think how it would have got in there? Is it possible that on a previous ride you got the wheel wet in the rain/puddles and possibly some moisture had not dried out before your pad set up session today?

Good question. The previous ride was 3 days earlier, dry weather.
Before this test ride with the pads, it rained during the day but the ground dried up.
The wheel side covers were open most of the time during the past few days to test 3D printed support prototypes.

Air humidity was high and when the issue occurred (80% according to Strava).
Damp air day but it is impossible that water remained from earlier or got in then. Some of the droplets are still on the cover 12 hours later, so it's probably not water.

Quote

The board version you have fitted is the same L20-42 version that I have had sent to me on 2 occasions as replacements after my "ECHAr" charger fail issues, but have both been solid in performance away from the "ECHAr" issue! Still, that does not rule out that you may have been unlucky and ended up with a weak/compromised board with less than perfect components or solder joints!

Yes I hope it's the reason. Good to hear your boards perform well!

Quote

The most current motherboard version though, appears to be L21-10 I believe (which supersedes the L20-42) as I have mentioned in recent posts elsewhere, but I'm unaware of the exact changes the latest version motherboard offers.

I would Not even attempt to ride it in it's current state and certainly would Not try any hacks to get the fans working or repair the failed component, as there is most likely more microscopic damage to the tracks within the board.

An 11 day old Sherman that failed whilst braking most definitely needs a New motherboard as the Only option to repair the wheel, but at least you will get the very latest L21-10 version as a replacement.

I'll be tempted to try if it takes a long time to get a replacement board since the wheel rides normally aside from the heat buildup
I would complete a serious stress test first before riding it further.

Quote

I wish you luck, but Leaperkim should see you right without issue may just take a little while is all.

Thanks, I hope they can ship a board directly here to reduce the delay. The shop owner contacted them already. But yeah it case it fails, that could damage the wheel further.

 

3 hours ago, Chriull said:

Temperature is reported from some IC on the motherboard. Like the gyro chip. So it is just some measurement of the average compartment air temperature. And the self-heating of this IC.

So for short and high burdens one will not see any real temperature rise but already has fried components on the board.

Definitely. On the graphs you can see the temperature climb pretty slowly. Even with some temperature inertia from the heatsink it can't reflect the temperature of critical component. At the same time, MOSFETs didn't die here.

Quote

Although in your case the motor driver mosfets seem not to be overburdened and fried, they are best saved by a phase/motor current alarm. There is no way they can stand some 160-220 A beside very short and seldom peaks.

Yes according to the graphs, it peaked beyond -220 A phase current when braking but only for very short peaks.
And estimated beyond 2*30A which are the fuses installed as well.
I was paying attention to the motor sound as @DjPanJan suggested, which only grunted on regen.

Graph with phase amps: https://docs.google.com/spreadsheets/d/1hl9nmW-gmc3nZK_v3b5UVDDIvjy6Oues5cN56jC9o20/edit?usp=sharing

Quote

Same for the battery current - more then 70A are enough to desolder the battery wires from the mainboard ... So here only short and seldoms peaks are possible without damages...

The design and the burdens are just borderline. 

I'll add amp alarms indeed.

 

3 hours ago, Planemo said:

Interesting circs. It almost seems like the fans weren't working from the get go after the rebuild....possible?

Good question, the wheel remained chill during the 3 previous rides which were after changing the tire, so not possible:
https://euc.world/tour/599713990197120
https://euc.world/tour/599741692703070
https://euc.world/tour/599781096613229
 

Thanks guys for looking this as well ^_^

Edited by supercurio
  • Like 3
Link to comment
Share on other sites

24 minutes ago, supercurio said:

Note: estimated battery current by EUC World

In this range for such durations battery current could indeed pose a problem, too! As many pictures of burned motherboards have a black hole were the battery wires were once soldered to the mb.

26 minutes ago, supercurio said:

(Almost?) Around 120A continously for some 10-20 seconds? 

Afair with such numbers (older?) GWs were about to die. But i have no idea how motor currents between gw and veteran correspond. There easily could be some factor...

43 minutes ago, supercurio said:

Yes I made sure to avoid the low speed hill climbing known issue, but I wonder if by riding the hilly street faster - and drawing more amps I could have created the same situation?

Looking at the graphs it seem you accelerated up the incline?

Going up a hill with different _constant_ speeds needs always about the same motor current!

Accelerations on an incline draws much more current!

49 minutes ago, supercurio said:

Yes according to the graphs, it peaked beyond -220 A phase current when braking but only for very short peaks

Which made battery voltage rising to 93V! Could be the dc/dc converters did not like these voltage spikes?

Or for whatever the blown ?shunt? Resistor was used for?

 

  • Like 1
Link to comment
Share on other sites

3 hours ago, Chriull said:

In this range for such durations battery current could indeed pose a problem, too! As many pictures of burned motherboards have a black hole were the battery wires were once soldered to the mb.

(Almost?) Around 120A continously for some 10-20 seconds? 

Afair with such numbers (older?) GWs were about to die. But i have no idea how motor currents between gw and veteran correspond. There easily could be some factor...

Looking at the graphs it seem you accelerated up the incline?

Going up a hill with different _constant_ speeds needs always about the same motor current!

Accelerations on an incline draws much more current!

Thanks for pointing that out.
Indeed there was acceleration going up this street in two parts, with a 90° turn in between.
There's not much flat part to accelerate beforehand and I stopped to adjust the pads and let the wheel cool down between the two parts.

The idea was to not stay at low speed / RPM, since the consensus in conversations so far was that the Sherman motor is inefficient at low speeds but just fine if rotating faster, as well as the board.

It shows that it's an invalid assumption 
It wasn't off-road, it wasn't slow, only riding the street next to where I live as I'm used to with my other EVs.

I took a new look at the board and I think your theory is correct indeed.

This capacitor is in sad state here:

PXL_20210708_145816201.thumb.jpg.e11a4f055c579d38b4fb0a317c34b7a1.jpg

 

And the board conformal coating, got vaporised where you predicted @Chriull (which I mistakenly for water):

PXL_20210708_145835168.thumb.jpg.22cc8dd38d3d2e6fa7044526d41df6bc.jpg

Quote

Which made battery voltage rising to 93V! Could be the dc/dc converters did not like these voltage spikes?

Or for whatever the blown ?shunt? Resistor was used for?

From this second look, I'm sure that this component got cooked by the high board temperature.
It seems more likely now that the failure was from the heat buildup rather than hard braking specifically.

So dammit, I managed to fry this board accidentally in less than 30s, within 1min of leaving home.
All that despite researching extensively how not to while waiting.

At least, it's lucky that the board didn't fail catastrophically - throwing me onto the pavement at 40 km/h, it would probably soon after.
This wasn't a stress test, I only meant to adjust the fit of my power pads.

This wheel needs better alarms and protection so bad.

  • Like 2
Link to comment
Share on other sites

57 minutes ago, supercurio said:

From this second look, I'm sure that this component got cooked by the high board temperature.

No. As it seems in your first picture this resistor R5? was about to desolder itself. And the case deformed by excessive heat.

This cannot happen by 'high" board temperature - this comes from serious overburden == high currents through this resistor. He got heated up somewhere very roughly around  200°C - depending on the used solder's melting point.

57 minutes ago, supercurio said:

The idea was to not stay at low speed / RPM, since the consensus in conversations so far was that the Sherman motor is inefficient at low speeds

That's nothing unique for the sherman - that's system imanent for actual EUC design.

Unique to the sherman is the immense amount of current the battery can deliver!

Quote

 but just fine if rotating faster, as well as the board.

Yes. But "?quickly?" accelerating up an incline can be much worse than going up an incline slowly.

Especially "just" 15% - if i get this right (15m height difference within 100m) this is "quite nothing" for an EUC!

You can ride this up and down all day long in ealking speed without really burdening the wheel...

57 minutes ago, supercurio said:

This capacitor is in sad state here:

Yes indeed - has to be replaced!

57 minutes ago, supercurio said:

And the board conformal coating, got vaporised where you predicted @Chriull (which I mistakenly for water):

Afaik this wires go to the charge plug? Imo the main battery power wires supplying the board are the 4 wires in the corner just beside the big capacitor.

So "strange traces" from the conformal coating could come from the hand soldering of the wires, or whatever? It's hard to recognice something with the reflections...

Or you mean this small holes in the coating? This seem to result from the tiny holes ("via" - used to connect both sides and maybe inner layers of the PCB)

Edited by Chriull
Typos
  • Like 2
Link to comment
Share on other sites

1 hour ago, Planemo said:

I still dont understand how many people push shermans hard and dont get problems though.

But yeah, that cap is mullered. As you say, I think you got very very lucky without a crash here.

I'm sure it's a matter of:

Incline:
It's harder to push the Sherman to its limits on the flat, I'm guessing most riders will reach the max speed before overburdening the board.
Probably a few have been close to that already but are just not aware of it.

If where you live is all flat, it might physically impossible to burn up the board that way.
As long as you add an unspecified amount of incline tho...

Pads setup:
That's what I'm used to for off-road on the 16X, and I replicated it on the Sherman:
At this angle, the front allows to get a lot more torque than most pads available on the market with a safe posture (knees bent)

pads-test-on-sherman.thumb.jpg.df92e5cacdff03f0b78d0519d040f3b1.jpg

Edited by supercurio
  • Like 1
Link to comment
Share on other sites

23 minutes ago, supercurio said:

It's harder to push the Sherman to its limits on the flat, I'm guessing most riders will reach the max speed before overburdening the board.

I'd say all - overburdening on the flat is not really possible.

Reaching max speed means overleaning and faceplant..

Or do you mean some set tiltback?

26 minutes ago, supercurio said:

Probably a few have been close to that already but are just not aware of it.

Maybe with continuous full acceleration/deceleration cycles...

  • Like 1
Link to comment
Share on other sites

2 hours ago, Chriull said:

No. As it seems in your first picture this resistor R5? was about to desolder itself. And the case deformed by excessive heat.

This cannot happen by 'high" board temperature - this comes from serious overburden == high currents through this resistor. He got heated up somewhere very roughly around  200°C - depending on the used solder's melting point.

Got it, thanks. I didn't realise a resistor would be part of the power delivery circuit.

Quote

That's nothing unique for the sherman - that's system imanent for actual EUC design.

Unique to the sherman is the immense amount of current the battery can deliver!

You're right, I was listening for motor grunting or pedal dipping - but none of that happened here, with the 24s10p packs happily delivering far too much current for the board's own good.

Quote

Yes. But "?quickly?" accelerating up an incline can be much worse than going up an incline slowly.

Especially "just" 15% - if i get this right (15m height difference within 100m) this is "quite nothing" for an EUC!

You can ride this up and down all day long in ealking speed without really burdening the wheel...

For sure. I'm still surprised how quickly and easy that happened with not that much incline.

Quote

Yes indeed - has to be replaced!

Afaik this wires go to the charge plug? Imo the main battery power wires supplying the board are the 4 wires in the corner just beside the big capacitor.

The wires where you see the coating was vaporized are the battery connection to the mainboard.
The one we've been guessing fires start from. This case seem to confirm the theory.

Quote

So "strange traces" from the conformal coating could come from the hand soldering of the wires, or whatever? It's hard to recognice something with the reflections...

Or you mean this small holes in the coating? This seem to result from the tiny holes ("via" - used to connect both sides and maybe inner layers of the PCB)

I don't know about the small holes either, I can try to capture it better.

Edited by supercurio
  • Like 1
Link to comment
Share on other sites

16 minutes ago, supercurio said:

don't know about the small holes either

Sorry, i did not express myself clearly - these are small throughholes in the pcb conecting different layers. So conformal coating entered this holes and by this the holes on the surface stayed.

 

  • Like 1
Link to comment
Share on other sites

As follow-up Leaperkim asked the shop for me to forward the motor serial number, and display pictures (I'm guessing, pictures of all advanced settings)

What I found, looking at the board again: 3 out of 4 battery wires started to de-solder themselves from the board.
A few seconds later, and that would have shorted one or both battery pack (then blow fuses)

I still can't believe how easily that can happen, without warning of any kind.

PXL_20210712_133042588.thumb.jpg.2e3054bb24f54589aa436ed420ac2587.jpg

  • Like 2
Link to comment
Share on other sites

15 minutes ago, supercurio said:

What I found, looking at the board again: 3 out of 4 battery wires started to de-solder themselves from the board.
A few seconds later, and that would have shorted one or both battery pack (then blow fuses)

Not sure about just blown fuse.

Maybe also battery wires shortening the cap, cap vaporizes, board gets a nice black hole in that area like in these pictures:

Imho the veteran direly needs the battery wires fixed to the board like the motor wires (screw on socket). Plus some of this fire/heat safe sleeves...

  • Like 2
Link to comment
Share on other sites

Just a quick note after a quick read through. The “water” might have come from the capacitor. Some compactors are filled with a very light oil that looks just like water when it leaks out. I have had this happen with a few new small 220 volt motor drivers this year. 

  • Like 3
Link to comment
Share on other sites

4 hours ago, Chriull said:

Not sure about just blown fuse.

Maybe also battery wires shortening the cap, cap vaporizes, board gets a nice black hole in that area like in these pictures:

That would have been the next step, no doubt.
The failures are pretty consistent.

At least this time we have a log that informs us of what are the limits of the board.

I'm thinking about modelling Sherman alarms in my EUC Toolkit app using the data gathered.

4 hours ago, Chriull said:

Imho the veteran direly needs the battery wires fixed to the board like the motor wires (screw on socket). Plus some of this fire/heat safe sleeves...

Ah yes, that would avoid these de-soldering themselves.

I wonder how many ride around with half-desoldered battery wires on their Sherman board.

  • Like 3
Link to comment
Share on other sites

4 hours ago, RockyTop said:

Just a quick note after a quick read through. The “water” might have come from the capacitor. Some compactors are filled with a very light oil that looks just like water when it leaks out. I have had this happen with a few new small 220 volt motor drivers this year. 

Thanks @RockyTop for mentioning this. There was no water indeed.

By the way since the capacitor is dead, there's no spark anymore when reconnecting the battery (I had to, to take some pics of the display for Leaperkim)
Nice that the capacitor failed by losing capacity and not shorting.

  • Like 1
Link to comment
Share on other sites

5 minutes ago, supercurio said:

Thanks @RockyTop for mentioning this. There was no water indeed.

By the way since the capacitor is dead, there's no spark anymore when reconnecting the battery (I had to, to take some pics of the display for Leaperkim)
Nice that the capacitor failed by losing capacity and not shorting.

Yes, I have been working with electronics for a very long time. I remember the old capacitors having oil in them and then for a long time they switched to a gel or paste.  Now seemingly recent they switched back to oil. 

  • Like 2
Link to comment
Share on other sites

  • 2 weeks later...

I got the replacement board today, Leaperkim shipped it from Hong Kong on the 19 and it arrived 2 days later in Sweden.
14 days total from the failure day, the time communicate with my dealer (EUCShop.lt who was on top of things, great service), a few back and forth with the manufacturer's support.

I was concerned that the Sherman would be unavailable for my upcoming riding holiday and possibly the whole summer season. Not at all.
While Leaperkim fails to program the board adequately in terms of alarms and protection, at least they can ship replacement boards timely.

So what about the board?
It's not a L21-10 as @fbhb anticipated but a L21-20.

board-full.thumb.jpg.30a73a9021b7047f0db6e79edf2ff5e6.jpg

 

Following @Tawpie and @Chriull's comments, I wonder if the soldering of the battery wires is sufficient.
What do you think?

soldering-1.thumb.jpg.b14b6beb54ddb1d066824e519547fb2f.jpgsoldering-2.thumb.jpg.0e9ada6de100d207776a4c21374727a8.jpgsoldering-3.thumb.jpg.3d290ea955241c0a5700a1bb3e3eb9e4.jpg

Then, the board came with a couple accessories, which I guess are likely to be damaged when a board need replacement.

board-accessories.thumb.jpg.619d067d664a4d3ebb2d69b5f5f2ce64.jpg

Edited by supercurio
  • Like 1
Link to comment
Share on other sites

One of the changes that we got from eWheels was that we need to change the 50 amp fuses to 30 amp. I am guessing that you had 50 amp fuses. Having 30 amp fuses might have prevented your board from melting. 

  • Like 2
Link to comment
Share on other sites

Looking closer, I'm not sure the one on the right is adequate.

I don't know enough about how multi layer board work to tell.
Would it help to secure the wire in place in case the solder start to melt with some high temp silicon, or only increase the risk of damage from heat?

soldering-4.thumb.jpg.b0f460a091c02a892f518ea2f77db83e.jpg

  • Like 1
Link to comment
Share on other sites

11 minutes ago, supercurio said:

I wonder if the soldering of the battery wires is sufficient.
What do you think?

The soldering is MUCH better, probably good enough to pass an IPC inspection. Yay!

EDIT: the black wire on the right has partial filling of the plated through hole, I think you 'want' 70% or more filled, that eyeballs at 50% but the bottom side fillet is good so I'd give it a pass. I wouldn't add anything. If things get hot enough to melt the solder that's a symptom of a larger problem that won't be helped by mechanical supports etc. Holding a plated through hole at solder melt temperatures for longer than a few seconds starts to cause physical damage so if it actually does reach solder melt temperatures, the joint IS going to fail.

Edited by Tawpie
  • Like 2
Link to comment
Share on other sites

12 minutes ago, RockyTop said:

One of the changes that we got from eWheels was that we need to change the 50 amp fuses to 30 amp. I am guessing that you had 50 amp fuses. Having 30 amp fuses might have prevented your board from melting. 

The wheel was from the latest batch (11 days old) and already had the 30 amp fuses.

In that instance, the board (partially) failed before the fuses blew.

8 minutes ago, Tawpie said:

The soldering is MUCH better, probably good enough to pass an IPC inspection. Yay!

EDIT: the black wire on the right has partial filling of the plated through hole, I think you 'want' 70% or more filled, that eyeballs at 50% but the bottom side fillet is good so I'd give it a pass. I wouldn't add anything. If things get hot enough to melt the solder that's a symptom of a larger problem that won't be helped by mechanical supports etc. Holding a plated through hole at solder melt temperatures for longer than a few seconds starts to cause physical damage so if it actually does reach solder melt temperatures, the joint IS going to fail.

Good to know what matter here :) thanks for the confirmation!

Next steps, putting this board in and working on alarms.
I plan to use EUC World phase current alarms first, and as soon as possible add also battery current alarms as well in my EUC Toolkit app, since it's a known point of failure.

  • Like 3
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...