Kingsong s20 Cut out FIRE (New York)

[Finn Bjerke]

1 minute ago, BatteryMooch said:

Still digesting what their report said though and reading the other technical responses about this.…great bunch of knowledgeable engineers here!

Is there a report somewhere ?     AN official Kingsong report ? 

[BatteryMooch]

Just now, Finn Bjerke said:

Is there a report somewhere ?     AN official Kingsong report ? 

Scroll up three posts. 🙂

[Some Canadian]

First off I just want to say I love to see this sort of response from a manufacturer, in house testing + a report? Love it, this should be normalized more.

As far as the report itself goes, it raised a few more questions for me, @BatteryMooch you already mentioned a lot of my points so I'll just put my "new" thoughts below =P
 

• Why is the over-current detection purely SW-based? I'll admit I'm not overly familiar w/ the design of BMS' in general (time to do some reading) but I find it odd that there is no over-current detection hardware (not fuse) built-in to the BMS. Relying on solely Software and a fuse seems odd to me.
• Why wasn't this "bug" discovered in QA? It seems like it was relatively easy for them to reproduce, and you'd think any reasonable QA around the BMS/battery systems would have caught this.

Overall I'll repeat I'm glad to see the information that's been provided (still digesting some of it, more digging to do on my end), but this just raises some more eyebrows for me. Particularly around the fact that they haven't yet mentioned a hardware fault / hardware revision to address the issue.

[Tawpie]

What's actually very interesting about this happening on the older firmware (before they increased the timing before over current shutdown was invoked). I had been secretly hoping the new firmware would show us where the limits on current draw were.

As it is, they exposed the fault with how the system re-boots. I wonder if the new firmware would have kept the battery connected long enough to blow the fuses? If that were the case (and I'm not saying or implying that the new firmware would have stayed connected long enough to blow the fuse), we might never have known there was a problem  with the way the system booted up into an existing fault condition.

[Tawpie]

11 minutes ago, Some Canadian said:

Particularly around the fact that they haven't yet mentioned a hardware fault / hardware revision to address the issue.

We're not done yet. They haven't talked about what started the sequence of events. I don't think they know yet... and sadly they may never know until it happens again with better data collection (and a not crispy critter motherboard). Hopefully they've got their riders out pretending they're U-Stride.

They also haven't disclosed the results of the third party evaluation of their battery and BMS. Although it's possible that the third party pointed out the current problem... just don't know.

Edited April 2, 2022 by Tawpie

[BatteryMooch]

3 hours ago, Some Canadian said:

Why is the over-current detection purely SW-based? I'll admit I'm not overly familiar w/ the design of BMS' in general (time to do some reading) but I find it odd that there is no over-current detection hardware (not fuse) built-in to the BMS. Relying on solely Software and a fuse seems odd to me.

Most BMS controller chips use just their firmware and current sensing resistors to detect and respond to overcurrent/short-circuit conditions by shutting off the FETs and/or blowing a “chemical” fuse. These fuses have current ratings that are too low for most PEV’s though. Many commercial BMS’ stop there.

There is something called “secondary protection” though in some good BMS’ that is hardware based. It’s typically for over-voltage protection, the biggest concern for keeping most commercial packs from going into runaway. It uses a separate dedicated chip to detect the problem and shut off the FETs or blow a chemical fuse in case the main BMS controller fails to do this. It wouldn’t be hard to do this for over-current/short-circuit conditions though.

Why don’t all BMS’ use this dedicated-hardware secondary protection? Cost and the extra PCB space needed. Some manufacturers feel that extra protection like this isn’t needed. Or perhaps that the cost of possible returns is lower than the cost of adding this feature? Though that ignores the hit to a company’s reputation that these kind failures can cause.

Edited April 2, 2022 by BatteryMooch

[Some Canadian]

46 minutes ago, BatteryMooch said:

...

There is something called “secondary protection” though in some good BMS’ that is hardware based. It’s typically for over-voltage protection, the biggest concern for keeping most commercial packs from going into runaway. It uses a separate dedicated chip to detect the problem and shut off the FETs or blow a chemical fuse in case the main BMS controller fails to do this. It wouldn’t be hard to do this for over-current/short-circuit conditions though.

Why don’t all BMS’ use this dedicated-hardware secondary protection? Cost and the extra PCB space needed. Some manufacturers feel that extra protection like this isn’t needed. Or perhaps that the cost of possible returns is lower than the cost off adding this feature? Though that ignores the hit to a company’s reputation that these kind failures can cause.

Ah gotcha, that does make sense then. My main experience with over voltage/current circuits is with MUCH smaller values haha (talking 5v/1A ranges) and I imagine that for high voltage/current situations, it's a bit more complex than "just place this protection chip" down (And even if it is that "easy", the parts are likely physically much bigger). Cost + board space is always a balancing act, and I can appreciate the engineering considerations that go into that. However all that being said, I would like to see more precautions taken with an EUC BMS, compared to say an e-bike BMS, since the failure states can typically lead to much worse outcomes.

At the end of the day this is a "niche" market, and there are always "growing pains" and faults that pop up (although some might say that some faults should never pop up). To me it's all in how the manufacture(s) react to these issues, and pivot/change moving forward. So will just have to keep my eyes on this . 

Edited April 2, 2022 by Some Canadian

[Paul A]

 

[Paul A]

 

[BatteryMooch]

@Paul A, thank you for posting that…very interesting.

Though I would tend to think that using the terms “PVC” and “fireproof” together really doesn’t work well. Not with a plastic that typically melts at around 100°C. 🙂

[chanman]

Yeah that stuck out to me too, maybe they meant waterproof? Or fireproofing in the sense of preventing fires from water ingress? Guessing just some translation issues there. 

[The Brahan Seer]

I think a possible hardware solution going forward is the installation of an isolation switch so in the event of a malfunction we can isolate the batteries from everything else very quickly

(As seen in electric cars). Should this be mandated for all wheels?

Just curious what would have happened if they managed to turn off the wheel as soon as it fell? Maybe nothing I suspect? Also shows the issue of having the power button so far into the wheel too. Not easy to reach quickly in these circumstances.

Edited April 2, 2022 by The Brahan Seer

[Forwardnbak]

So fix is software based only?

no redesign of anything? 
 

S22??
 

I’m not sure about all this just yet. 
 

[chanman]

4 minutes ago, The Brahan Seer said:

Just curious what would have happened if they managed to turn off the wheel as soon as it fell? Maybe nothing I suspect? Also shows the issue of having the power button so far into the wheel too. Not easy to reach quickly in these circumstances.

Hitting the power button definitely is not going to help with the board already smoking, it's just a way of requesting the board to go into lower power mode and turn off the balancing features. If you were to have an emergency switch or button for cutting off the batteries without going through a software layer that would have potentially helped, but that then becomes a liability while riding the wheel.
 

[fryman]

53 minutes ago, The Brahan Seer said:

I think a possible hardware solution going forward is the installation of an isolation switch so in the event of a malfunction we can isolate the batteries from everything else very quickly

(As seen in electric cars). Should this be mandated for all wheels?

Just curious what would have happened if they managed to turn off the wheel as soon as it fell? Maybe nothing I suspect? Also shows the issue of having the power button so far into the wheel too. Not easy to reach quickly in these circumstances.

There is the handle kill switch.

[supercurio]

Okay so this first address is really not satisfying at all, with holes and issues.
I am frankly stunned that this looked good enough for them to upgrade the name to S22.

However I am grateful and appreciative that they share details of the process with us, since it allows to identify flaws in it easily.

Invalid fuse test

In the Tests video (with timestamp)

In this test, the operator blows the fuse with an electrical short (software protection disabled).
Then the operator instantly stops the short manually.
Several attempts confirm that since the fuse blew and short circuit was interrupted prior, it is not possible to re-create a short - as expected.

However this test misses the point. Why?

• When a short occurs, it is continuous.
• Batteries generate DC voltage and currents: it means that the short has an ability to arc. Although @Tawpie taught me that the arc tends to vaporize the conductors involved rapidly.
• In the NYC fire, the short in the motherboard was most likely continuous, it is a different scenario than demonstrated
• This fuse is rated at up to 60V DC. Not 126V DC. Which means it might arc until the circuit is interrupted

What Kingsong needs to do

Kingsong must show us the operator creating a continuous short - also with software safety disabled, to test if the hardware BMS fuse will be enough to prevent a fire in case of

• another firmware bug
• blown MOS on the BMS, possibly caused by over-voltage condition, like hard braking near full battery

With this 100% continuous short simulating blown MOSFET on the mainboard, KingSong would need to demonstrate that the fuse interrupts the short and that the pack does not catch fire.
Why? Because the fuse didn't work as expected during the last incident.

This test would need to be replicated in two environments:

1. outside the battery box in order to see clearly what happens
2. inside the sealed battery box, like in a wheel in order to replicate real usage conditions accurately

 

KingSong assumes sensorless backup does not work

Quote

After analysis it may be that the system detects an error of hall sensor signal during the riding, resulting in failure to output power.

In all text/image/video marketing material, KingSong has boasted about the sensorless backup mode, to prevent issues or a crash in case of hall sensor failure, which has been a concern on S18.

Yet the first assumption of their engineer is that an error in hall sensor signal results in failure to output power.

Is this sensorless controller capability even existing?
Is it planned for later?
Is it buggy or malfunctions like the other safety mechanisms?

I find suspicious that it's the first thing that comes to mind.
Best case, sensorless control might not work well enough to stabilize the wheel at lower speeds.

 

KingSong has not reproduced the fire scenario

KingSong has found a software bug, and might have fixed it (hard to tell for sure without further regression and real-life testing, which takes a longer time).
By all means, this rushed bug-fix might introduce cutouts during normal riding conditions introducing current spikes.

So okay, KingSong addressed 1 issue that can be identified in a controlled environment. What they have not identified nor addressed:

• Why the mainboard burned
• Why the BMS fuse didn't work (explanation above)
• Why fire propagated from one pack to the second in a few seconds.

 

In order to claim to fix a problem, you must show that you were able to reproduce it first.

Hopefully there will be a more complete address after analyzing the mainboard.
Since 2.19 lists "Optimized the log record of the motherboard", it's uncertain how much data was captured in the S20 fire, although KingSong expected to possibly be able to recover some from the incident despite sad state of the mainboard.

 

Overall, my main concern is the apparent tension between Business/Marketing and Engineering.
I find ludicrous that after a fixing exactly 1 bug (needs confirmation with more testing), KingSong felt that it was legitimate to re-brand the whole product to S22.

This is only the beginning of the efforts to make this wheel a safe vehicle.
What will be the final product ready to match our expectations. S28? S42?
 

Edited April 2, 2022 by supercurio

[BatteryMooch]

54 minutes ago, supercurio said:

When a short occurs, it is continuous.

Fantastic analysis, thank you for posting!

A thought I had…

While a continuous short can definitely happen I think intermittent shorts are a very real possibility and also need to be thoroughly tested for.

This is a high vibration environment and abrasion and other vibration/movement induced wear on insulation or snapping off of components could cause intermittent short-circuits as things bounce around. IMO, these can be worse than a continuous short in some ways. A continuous short will often clear itself due to the burning out of the conductive path. Of course, it might not do that before a cell is forced into runaway but it does happen.

The huge current flow during a short-circuit will create a large inductance-created voltage spike when the current flow stops. Multiple short-circuits can easily cause breakdown and destruction of different components, even TVS diodes placed to absorb these spikes. But there are no TVS diodes in this EUC (with a working voltage of 126V and a 150V FET Vds rating) as a 130V SMCJ TVS diode (the lowest voltage rating you could use) clamps at over 200V. Hmm…wondering what transient suppression they are using.

Because of the short duration of these intermittent shorts the conductive paths might not be be cleared but the huge voltage spikes would continue. This additional damage could cause a failure that would accelerate or even cause failure of the pack.

These short term events (short circuits) can also play havoc with power integrity for the controller chips in the ESC and BMS, causing brownouts and repeated forced reboots. This can leave the circuit in undefined states and or cause lockups or unintended firmware behavior, leading to all sorts of problems that could destroy the pack.

Edited April 2, 2022 by BatteryMooch

[Forwardnbak]

The renaming thing just feels yuck 

feels… yuck 

[supercurio]

1 hour ago, BatteryMooch said:

A thought I had…

While a continuous short can definitely happen I think intermittent shorts are a very real possibility and also need to be thoroughly tested for.

This is a high vibration environment and abrasion and other vibration/movement induced wear on insulation or snapping off of components could cause intermittent short-circuits as things bounce around. IMO, these can be worse than a continuous short in some ways. A continuous short will often clear itself due to the burning out of the conductive path. Of course, it might not do that before a cell is forced into runaway but it does happen.

Thanks for all the details, I am actually still puzzled how a short could extract enough current from the battery in order to create ignition in the first place.

Would you have some theories on that?
Is it the heat from the red hot nickel strips which is transferred to the cells and trigger their thermal runaway?

1 hour ago, BatteryMooch said:

The huge current flow during a short-circuit will create a large inductance-created voltage spike when the current flow stops. Multiple short-circuits can easily cause breakdown and destruction of different components, even TVS diodes placed to absorb these spikes.

Because of the short duration of these intermittent shorts the conductive paths might not be be cleared but the huge voltage spikes would continue. This additional damage could cause a failure that would accelerate or even cause failure of the pack.

These short term events (short circuits) can also play havoc with power integrity for the controller chips in the ESC and BMS, causing brownouts and repeated forced reboots. This can leave the circuit in undefined states and or cause lockups or unintended firmware behavior, leading to all sorts of problems that could destroy the pack.

 

Understood. it further illustrates that a software-based BMS safety is not sufficient for the worst-case scenario.
Although it would catch most of the shorts.

I understand better in the written analysis the importance of the reboot and timing condition.

 

Reading the pages 2 and 3 again, bring a sense of confusion:

Mainly, in 2/ they explain that the fuse is incapable of stopping the short, and that when the software protection malfunction the pack catches fire.
I read that as a confirmation that hardware changes on the pack and BMS are required, since KS own experiment demonstrates that the hardware fuse in ineffective in preventing a fire.

And in 3/ they explain more about how the software protection (using the BMS MOS) fails.
That's another confirmation, in writing that hardware changes are needed to prevent a fire escalation in case software+MOS protection fails.

In short, most of the written report is clear enough.
What is confusing however is

• Videos showing invalid tests
• PR-driven early rebranding, despite the absence of mention of any of the hardware changed required
• The omission that while a firmware update is expected soon, it is insufficient to make the wheel safe, since none of the known or TBD hardware issues were addressed

So let that be clear to everyone. Kingsong is not done here. All issues were not addressed. 

KingSong stating that "riders must update the software to ride safely to the latest version to ensure a safe experience" is false.

Edit:
To add to the confusion, their video show that the fuse blows indeed, contradicting their own report.
It might mean that packs also get a hardware update, although there is no mention of it in the hardware report (only software)

Edited April 2, 2022 by supercurio
more confusion

[BatteryMooch]

30 minutes ago, supercurio said:

Thanks for all the details, I am actually still puzzled how a short could extract enough current from the battery in order to create ignition in the first place.

Would you have some theories on that?
Is it the heat from the red hot nickel strips which is transferred to the cells and trigger their thermal runaway?

Yes, that’s what I think too. While a “hard” short can send a cell into runaway it’s not easy with standard (non-LCO chemistry) li-ion cells from the major manufacturers.

Based on their mention that the nickel gets red-hot I also think that heat transfers to the cells and sends one or more into thermal runaway. But…what the frakk’ is going on?!? How can a “tested” BMS allow something as ridiculous as this to happen? 🤬 

I am confused and angry about so many things with all this. If it’s not the hot nickel causing the runaway then I have no idea what the heck is going on. As you mentioned, KS’ report answers some questions but raises even more IMO.
 

35 minutes ago, supercurio said:

So let that be clear to everyone. Kingsong is not done here. All issues were not addressed. 

Amen!

[Tawpie]

46 minutes ago, supercurio said:

Is it the heat from the red hot nickel strips which is transferred to the cells and trigger their thermal runaway?

That, plus internal heating of the cells themselves can easily get the cells to thermal runaway. The red hot ends are welded to the battery terminals... great heat transfer!

46 minutes ago, supercurio said:

Mainly, in 2/ they explain that the fuse is incapable of stopping the short, and that when the software protection malfunction the pack catches fire.

I read it the other way around... the software malfunction (reconnecting the battery upon reboot and not locking it out) is what kept the fuse from blowing (the fuse is the final hardware failsafe, it's there in case the SW or MOSFET can't or won't disconnect power). The fuse can and was demonstrated to blow if the MOSFET disconnect is completely disabled. I understand them to mean that the problem was that the short circuit didn't stay in place long enough for the fuse to melt (the fuse had time to cool down). Think of the SW and the BMS MOSFET as a resettable circuit breaker. If you keep turning the breaker back on over and over, your wires will overheat and your house will catch on fire. But an inline fuse might never blow. The revised firmware won't turn the breaker back on all by itself, it'll force you to take some other action before it'll reset the breaker. Hopefully you won't connect to charger, have it trip off, then connect to charger, have it trip off, then connect to charger over and over real fast. If it doesn't work the second time, STOP MESSING WITH IT and service it! (this is true for most things, right?)

On page 3 they mentioned a motherboard firmware bug that continued to apply power to the motor after the presumed hall sensor failure. I don't know if this bug falls into the V.19 paragraph D "other known bugs" or not. I don't personally condone sending out firmware with known bugs, but this was demo equipment and we all know that all software is pretty much guaranteed to have known problems when it ships. It will absolutely have known problems the first time it leaves the company's control (this was sort of a beta test—first truly uncontrolled use of the software).

Edited April 2, 2022 by Tawpie

[supercurio]

34 minutes ago, Tawpie said:

That, plus internal heating of the cells themselves can easily get the cells to thermal runaway. The red hot ends are welded to the battery terminals... great heat transfer!

Thanks, it helps to understand the usefulness of thicker nickel strips.

 

34 minutes ago, Tawpie said:

I read it the other way around... 

Frankly I'm not sure after reading it one way then other multiple times.

However my understanding for now is that the explanation describes conditions introducing the loop which causes the MOS to not remain in the OFF state, although...

Easy to get confused when it contains sentences like these (missing a negation)

Quote

At this phase of a bug in the BMS the MOS should turn off to protect all components, but that did happen.

Followed by 

Quote

and the time between ON and OFF states were so short that it did not blow the fuse

That sounds good, right - meaning not enough current?

Quote

hence the constant power output from the battery led to the eventual fire incident

In a scenario where the BMS switches between "ON/OFF" states there's not "constant power output", this is contradictory.
And does that mean that certain current are enough to get the nickel strip red hot which would ignites the cells, but still not enough to burn the fuse?

If a combination of average current and ON/OFF state timing can lead to this result, can we say that it is the wrong fuse for the job.

 

34 minutes ago, Tawpie said:

On page 3 they mentioned a motherboard firmware bug that continued to apply power to the motor after the hall sensor failure. I don't know if this bug falls into the V.19 paragraph D "other known bugs" or not. I don't personally condone sending out firmware with known bugs, but this was demo equipment and we all know that all software is pretty much guaranteed to have known problems when it ships. It will absolutely have known problems the first time it leaves the company's control (this was sort of a beta test—first truly uncontrolled use of the software).

Right, that's another interesting one.
This bug is likely to get the board self-destruct. It's true that the board is supposed to detect bad or lack of hall sensor data, disable motor control and beep. The fact that it's not might reveal an incomplete sensorless implementation since in that case it should not stop driving the motor but switch to sensorless instead.

Edited April 2, 2022 by supercurio

[Tawpie]

24 minutes ago, supercurio said:

If a combination of average current and ON/OFF state timing can lead to this result, can we say that it is the wrong fuse for the job.

Yes, absolutely, that's fair.

Keep in mind that an ordinary fuse is a horrible way to try to protect things. That's why we have ground fault interrupters and circuit breakers in our homes now, and not the screw in fuses. In my opinion, you can't rely on a 'fuse' as a primary failsafe... because it's just not very good at it. As is, bad software found a way to defeat a hardware mitigation—not that the hardware mitigation has anything to crow about.

A fuse as a last resort makes sense. It's like a weak link. But it is a last resort... you've now experienced two cascading failures and need the third to work. I'm glad it's there even if it didn't work in this case because it's clear that two failures can happen. If an unfortunate software design or implementation choice hadn't been made, we'd be focusing on a controller failure and assplant.

I'm willing to wait, but I do want KS to continue this open attitude and let us know what they think actually caused the assplant... and what if anything they're going to do about it. It's not anywhere near as bad as a fire, but a controller failure isn't particularly welcome either, even if I don't plan on riding with the NYC crew. Get that sorted and it's on to the next thing. I'm hoping that'll be something really serious like "the paint fades in the sun".

Edited April 3, 2022 by Tawpie

[BatteryMooch]

@supercurio and @Tawpie, it just occurred to me….

The hot nickel doesn’t have to heat up the cell enough to force it into runaway. If the nickel is pressing against the top of the cell then it could easily melt the plastic wrap and top ring insulator (if plastic, not paper) and directly short-circuit the cell. Perhaps unlikely but it’s not impossible.

Wouldn’t happen though IMO if paper rings were added when the pack was built.

[supercurio]

@BatteryMooch good thing we have a battery assembly videos! It's like they shared it with us for review 

It looks like your theory holds, although I'm not expert enough to tell for sure.

What do you think?