Sherman software alarms: how to model a wheel's failure modes?

[supercurio]

Hi all!

After getting a replacement board following the early half death of the original one, I'm well aware of how easily it is to burn one up and would prefer preventing any further occurrence, for myself and hopefully other Sherman owners as well.

So here's a plan:

Step 1: EUC World Alarms

A few topics discuss this already, with @Rawnei landing on 100A phase current alarm. It's not a peak alarm however so it's smoothed over time - I don't know how.

That might be a start for the next few days.. then,

 

Step 2: Implement phase current & battery current alarms in EUC Toolkit

I made this experimental app called EUC Toolkit, which gets realtime wheel and computed data from EUC World and makes use of that -currently only to control LED strips.

Now, much more can be done with this data.
From threshold to algorithms, or even real-time inference of machine learning trained models. Why not!

 

Step 3: Implement these alarms in a micro-controller attached directly to the mainboard

Once figured out and tested well enough software alarm implementations, they can be rewritten to run on a NRF52x/ESP32 like MCU board with bluetooth capability for setup, which receives the wheel data on a serial port, like the one going to the Bluetooth LE controller or possibly to the display.

This board would be connected to the existing beeper or its own and have low latency alarms, not depending on a phone + app + bluetooth connection.
More work and requiring some hardware installation, but more reliable.

 

How to help getting the project started

Logs

I'm interesting in wheel logs. To be more specific: wheel logs where you thought that the wheel could fail here but didn't.
Examples:

• The motor was making unhappy noises but you made it up this hill? Great!
• The wheel got over-powered when accelerating or braking.
• You were riding the beeps, or past the beeps. Maybe it went well, maybe you crashed. Show the log.
• Your Sherman overheated 
• Your board burned: got the magic smoke, possibly some fireworks and you recorded all that? I absolutely want this log.

 

In each of these case, I would suggest you check the status of your Sherman board after these instances. It might have a damaged cap or maybe the battery wires started to de-solder.

Expertise

I'm a software developer and will be happy to implement all sort of algorithms for this, even if it seems a bit complex.
I have a little experience designing and training ML models but not that much.

What I don't know is electrical engineering, electronics engineering, designing PCBs, the physics and calculations behind how a board and its components fail.
So I need help with that. Fortunately a few of you have this type of expertise, and its' an opportunity to put it to use for our EUCs in this project.

 

Project requirements

Everything will be open sources, so any of the code and findings can be re-used by other apps, and by manufacturers in their own firmware.

The goal is to increase reliability and safety of our wheels, and make this as open access as possible.

 

What about other wheels?

The Sherman is unfortunately not the only wheel capable of burning its own mainboard, as @Jack King Song recently illustrated in video.
It's a good first candidate however since it combines very high power with very little safety or alarms in programming, aside from speed vs voltage and low battery alarms.

This project might be useful for a few more wheels however in order to increase the desired safety margin desired.

 

Thanks for your attention! I can't guarantee success, but I need this project so I'll try.
I be grateful for any help also.

 

[Rawnei]

Nice initiative, regarding my findings that 100A current alarm worked well in EUC World I'm starting to think that either the current alarm behavior or the input values it's based on might have changed in later versions of EUC World, I have not been able to trigger the alarm even with very heavy acceleration such as I used to be able to so I don't know if 100A is a good baseline anymore. For example the other day I did a fast acceleration to 71kmh GPS speed so fast that Sherman gave me the fast beeps then slow beeps before EUC World even gave me any speed alarm, no current alarm was triggered, graph shows current around 55A at 92.3V, safety margin 13% (which I believe is not accurate in EUC World at least not for Sherman).

Looking at my past tours and the graphs you reported from when your wheel almost burned it seems more normal with an occasional sustained current around 50-55A with the rare peak at around 60A, so perhaps sustained current alarm in the ballpark of 50-55A is a good baseline to start with and then do some tests to see how it behaves?

Perhaps @Seba can chime in here with some observations and wisdom.

[supercurio]

1 hour ago, Rawnei said:

Nice initiative, regarding my findings that 100A current alarm worked well in EUC World I'm starting to think that either the current alarm behavior or the input values it's based on might have changed in later versions of EUC World, I have not been able to trigger the alarm even with very heavy acceleration such as I used to be able to so I don't know if 100A is a good baseline anymore. For example the other day I did a fast acceleration to 71kmh GPS speed so fast that Sherman gave me the fast beeps then slow beeps before EUC World even gave me any speed alarm, no current alarm was triggered, graph shows current around 55A at 92.3V

Maybe the 100 A was for phase current, but has been replaced by the estimated battery current data source.

It would mean that all the alarms set in a previous EUC World version (before the introduction of estimated battery current) are now broken.

If the 55 A estimate is accurate, it's not that far from the 2x30 A limit of the fuses and close to the limit of the board as well.

Question is: how long can that be sustained? That's what I'd like to model.

Fortunately, the wheel data available and recorded in logs still contains phase current as provided over Bluetooth, so even if EUC World doesn't support phase current alarms anymore, they can be reimplemented back to EUC Toolkit.

Quote

Looking at my past tours and the graphs you reported from when your wheel almost burned it seems more normal with an occasional sustained current around 50-55A with the rare peak at around 60A, so perhaps sustained current alarm in the ballpark of 50-55A is a good baseline to start with and then do some tests to see how it behaves?

Yes, that's where the idea of looking at many logs comes from to establish a baseline of safe, identify where it gets dangerous and figure out where things will fail.

It's a good start to look at our own previous rides that way, then further data analysis will be necessary to get more insights, especially regarding current over time.

Edited July 21, 2021 by supercurio

[fbhb]

5 hours ago, Rawnei said:

Nice initiative, regarding my findings that 100A current alarm worked well in EUC World I'm starting to think that either the current alarm behavior or the input values it's based on might have changed in later versions of EUC World

@Rawnei you are correct that later versions of EUC world have changed the way current alarms behave, hence your 100A setting no longer being triggered! 

As from version 2.6 @Seba implemented changes specifically to how GotWay/Begode and Veteran wheels need to be set up for current alarms to work.

Below is a quote from the EUC World blog:

"Changes in EUC World 2.6

While developing EUC World, I had been wondering for a long time if something could be done to improve the reliability of the current and power readout. Thanks to user support, it was possible to purchase a special logging device to record the current flow in the monocycle’s battery circuit while riding. Thanks to test rides and mathematical analysis of collected data, including correlations between speed, battery current and motor phase current, I was able to develop a formula that can be used to approximate the current in the battery circuit.

Therefore, starting from version 2.6, the current displayed in the EUC World application will be a value much closer to reality than before. This will not only allow a more accurate assessment of the performance of the unicycle, but also will allow an approximate measurement of energy consumption while riding. The current value displayed so far will be presented as a separate parameter: motor phase current.

This is, of course, just the beginning. As new data will be acquired, the algorithm will be improved. But already now this change should make it easier for you to use EUC World application by providing more reliable information about battery current and engine load in Gotway, Begode and Veteran wheels

Important note

This change affects the operation of current alarms in EUC World application. If you are a Gotway, Begode or Veteran wheel owner, check the current alarm settings for your wheel in EUC World app. The previous settings may be too high and cause the alarm not to be triggered. A safe setting for a unicycle equipped with a 100.8 volt battery is 25 amps, for an 84 volt battery it will be 35 amps. Of course, fine tune these values so they will work best in your case."

Edited July 22, 2021 by fbhb

[DjPanJan]

@supercurio i can send you many logs but where to upload. Is real question. 

to e-mail or some cloud. My sherman is V2 fat rim red buttons logs maked on 1.0.56 FW medium pedal mode PA +1 rider 🐷 weigt 260lbs+  (now i am on 1.0.58 but not ride this new FW yet)

6 logs is here: https://easyupload.io/m/f6bk74     (expire in 7 days)

EUC data 2021-07-19 191600.csv this one is focused for motor vibrating/grunding test. Every hill in this LOG is motor pain state i make this test becasue want test my watch vibrate alarm triger. And this day weather was "cold" temperature here ideal for testing.

[Rawnei]

8 hours ago, fbhb said:

This change affects the operation of current alarms in EUC World application. If you are a Gotway, Begode or Veteran wheel owner, check the current alarm settings for your wheel in EUC World app. The previous settings may be too high and cause the alarm not to be triggered. A safe setting for a unicycle equipped with a 100.8 volt battery is 25 amps, for an 84 volt battery it will be 35 amps. Of course, fine tune these values so they will work best in your case."

Ah I see, thanks, 25A seems a bit on the low side though according to graphs, have to experiment again.

[fbhb]

40 minutes ago, Rawnei said:

Ah I see, thanks, 25A seems a bit on the low side though according to graphs, have to experiment again.

Yes, 25A was too low in my use case also. IIRC, I started at 35A after upgrading to 2.6 but quickly increased in increments to around 50A-55A with some trial and error.

[DjPanJan]

Eucworld 2.7 Peak curent 65 A curent 40A safety "gap" 20% temperature alarm 65 celsius.

[supercurio]

3 hours ago, DjPanJan said:

6 logs is here: https://easyupload.io/m/f6bk74     (expire in 7 days)

EUC data 2021-07-19 191600.csv this one is focused for motor vibrating/grunding test. Every hill in this LOG is motor pain state i make this test becasue want test my watch vibrate alarm triger. And this day weather was "cold" temperature here ideal for testing.

Thanks @DjPanJan for sharing the logs, I downloaded them all and started to look at them as well.

Immediately it showed at least two approach on how to use the data:

1/ Use it as "all safe"

Since the wheel survived and the rider didn't crash by overlean or overpower, it essentially means all this is safe.

2/ Tag the data

As you mention for "EUC data 2021-07-19 191600.csv", there are parts where the motor is in uncomfortable territory (I'm guessing at lower speed & high phase current)

However without tags on where/when to look at and describing the situation, and a good way to visualise all the relevant key data point it is difficult to make sense of it.

I started to look into which tool would help here but have not found yet.
This was helpful tho: https://handsondataviz.org/

[DjPanJan]

http://wheellogviewer.net/ you can use this site for finding "interested parts" and filter. 

Normaly i use graphs form euc.world site find timeframes where this situation was. And after this i oriented in log file by time. (and i know map/places where i ride and inclines is by map)

But is imposible inject log into graph on eucworld site anymore.

Next rides/days i be testing 1.0.58 and my plan is make small eucworld "tours" just this style. Start tour under hill/incline and on top hill/incline stop tour this make some easy oriened LOG and tours for finding "our" goal.

Or make tours like loop with incline but i no prefer loop i want give my sherman time to cooldown after stugling time becasue my 🐷 weight.  

 

[supercurio]

@DjPanJan yes I used http://wheellogviewer.net/ also, but I found a few features missing for this purpose:

• show estimated battery current and motor phase current
• highlight / tag time periods
• share visualisation easily online

Maybe something using Jupyter Notebooks or Plotly Dash would work for this data science use case

[Chriull]

17 hours ago, supercurio said:

Implement phase current & battery current alarms in EUC Toolkit

Specific fixed current alarms need the rider to decide the urgency to react by weighting the alarm rate, duration, etc...

Imho that's why Seba did

17 hours ago, supercurio said:

It's not a peak alarm however so it's smoothed over time - I don't know how.

The value responsible for temperature rise is dissipated power. For "fixed" resistances as wires, mosfets (switched on),... this value is proportional to the square of the current.

Interesting should be some medium uration root mean of this squared current values and some shorter duration.

These values are also known as rms (root mean square) == effective current/voltage. For example the rms value from some alternating current is the value leading to the same power dissipation at a resistor (heating wire, lamp,...) as a constant current.

So exactly the value you want - with phase current to estimate the effect on the mosfets and motor wires, with battery current for the battery wires. For veterans battery current could maybe be of interest with his 10p configuration - for all other wheels battery current is of no interest for burden/overheating.

One has to experiment with the durations for these rms values and the limits for specific wheels.

 

[supercurio]

5 hours ago, Chriull said:

Specific fixed current alarms need the rider to decide the urgency to react by weighting the alarm rate, duration, etc...

Yes that's a good point, I also thought different or evolving sound depending on the severity of the alarm. Can be frequency of the beeping, frequency of the tone, or both at the same time.

I think a few users would like to be able to tell if they're are 70%, 90% or 99% of the hardware limit.

However as initial design target I will focus on the use case of riders who stop to push as soon as it beeps, which is reasonable for the Sherman given the amount of speed and power available already.

5 hours ago, Chriull said:

Imho that's why Seba did

The value responsible for temperature rise is dissipated power. For "fixed" resistances as wires, mosfets (switched on),... this value is proportional to the square of the current.

Interesting should be some medium uration root mean of this squared current values and some shorter duration.

These values are also known as rms (root mean square) == effective current/voltage. For example the rms value from some alternating current is the value leading to the same power dissipation at a resistor (heating wire, lamp,...) as a constant current.

So exactly the value you want - with phase current to estimate the effect on the mosfets and motor wires, with battery current for the battery wires. For veterans battery current could maybe be of interest with his 10p configuration - for all other wheels battery current is of no interest for burden/overheating.

Ah yes so there would be at the minimum:

• mosfets
• motor wires
• battery wires
• this resistor
   

Then after accounting what creates heat, we model what absorbs or removes it: ambient air in the case, board itself, heatsink thermal inertia, airflow, fans?

It looks like the only thing connected to the heatsink are the mosfets themselves. The fans are attached to the heatsink and will circulate, maybe somewhat cool the air inside the top chamber at the front of the board.

At the rear of the board, there will be almost no airflow - where the battery wires are connected.

Do you know how to model heat inertia and cooling, or is it best to infer from direct measurements?

5 hours ago, Chriull said:

One has to experiment with the durations for these rms values and the limits for specific wheels.

Yes, how would you suggest to experiment here without too much destructive risk.

For the Sherman, what I can think of right now is placing a bunch of temperature sensors on various hot spots of the board, and record how quick temperature rise and fall in various scenarios.
Would recording the board with a thermal camera (like FLIR) give valuable insight?

[Chriull]

2 hours ago, supercurio said:

Do you know how to model heat inertia and cooling, or is it best to infer from direct measurements?

I'm just used to the normal static simple models as used for heatsink design (like described in https://www.fictiv.com/articles/heat-sink-design-guide - did not read it, but scanning it had the right formulas...)

These model is only for unlimited convection. In a compartment as conmon with EUCs the ambient tempurature rises and by this the cooling effect of the temperature difference decreases. The enforced air movement could over/under or just compensate for this. With local differences?

Would need some finite elements simulation - but this would be way overdone conplexity wise.

2 hours ago, supercurio said:

Yes, how would you suggest to experiment here without too much destructive risk

Take logs of burdenous rides and create rms of the motor current with different time frames. Or less weight for older values (cooling).

Over time one should reach sensefull values?!

2 hours ago, supercurio said:

the Sherman, what I can think of right now is placing a bunch of temperature sensors on various hot spots of the board, and record how quick temperature rise and fall in various scenarios

This could help finding the right cooefficients. You'd just have too look for the most direct and efficient coupling of the sensors to the heat sources. So one sees (some) peaks and dynamic changes. 

2 hours ago, supercurio said:

this resistor

I could not really identify this resistor - but could be some quality control issue?