Kingsong s20 Cut out FIRE (New York)

[Paul A]

An inline fuse.

Would be fantastic if that is all that is required.

 

And if it was the BMS turned off.....

Licking the packing tape would be too good for that developer for all the heartache caused.

 

[Paul A]

https://www.washingtonpost.com/wp-srv/national/longterm/space/stories/orbiter100199.htm

October 1, 1999

NASA's Mars Climate Orbiter was lost in space last week because engineers failed to make a simple conversion from English units to metric, an embarrassing lapse that sent the $125 million craft fatally close to the Martian surface, investigators said yesterday.

_________________________________

Software error.

It happened to NASA and cost $125 million.

________________________________

 

https://raygun.com/blog/costly-software-errors-history/

Jan 26, 2022

The mere mention of a serious software error can strike fear into the heart of any developer, project manager or tech leader.

The wrong error in the wrong system can be incredibly expensive and difficult to resolve, not to mention humiliatingly public. Catastrophic software errors are mercifully rare these days, but the potential for chaos, PR disasters and spiraling costs still remains.

Error monitoring solutions have become more and more of a necessity in recent years, but plenty of businesses still view monitoring as an optional extra, despite the fact that poor quality software continues to cost US organizations over $2 trillion annually.

[Planemo]

Strong, IP rated sealed aluminium battery boxes are absolutely the way forward. I'm amazed anyone would think otherwise, and even more surprised that people moaned about old skool protection like skinny heatshrink or flimsy plastic covers but then bitch when they are given aluminium enclosures. I guess you never can please some people.

Look, aluminium boxes are not the problem here. Power management is.

My Bosch ebike battery is fully encased aluminium and packed so well there is no movement of anything anywhere. I trust the pack 110%, and have less than zero concerns storing it in my house.

Its all about power management, not flapping about wanting the cells to blow apart so they dont set fire to their neighbours., which is a bit like locking the gate after the horse has bolted.

[supercurio]

5 hours ago, Rawnei said:

I see this narrative pushed over and over, the wheel wasn't even on latest FW version as confirmed by the owner and it's obvious that failsafes that need to be there regardless of performance didn't work, so can we please not do this?

We already saw this narrative starting to be deployed by KingSong yes, that they think they know.

But! At least the time they published their first 2 statements, it doesn't mean they know. Only that they think they do - big difference.

[The Brahan Seer]

3 hours ago, Planemo said:

Strong, IP rated sealed aluminium battery boxes are absolutely the way forward

I totally agree and the following shows how Bosch do it, using great seals and one way pressure valves. However Kingsong and ourselves can take heart though,  even they make mistakes in assembly and reading the comments their batteries aren't without issues as shown below: 

 

[Robse]

This Bosch battery is one piece of crap !

[Planemo]

1 hour ago, The Brahan Seer said:

I totally agree and the following shows how Bosch do it, using great seals and one way pressure valves. However Kingsong and ourselves can take heart though,  even they make mistakes in assembly and reading the comments their batteries aren't without issues as shown below: 

 

I saw that vid when it was released, among many other vids at the time. Despite the complexity I still went ahead and bought my Bosch powered Haibike.

No battery is going to have a zero % failure rate, and especially not when zillions of them have been sold.

The only issue I have with Bosch is that the entire system is locked down and theres no chance of me re-celling it, but I will wear that for the peace of mind. Specifically, out of the huge number of them in service, I'm not aware of one ever having caught fire. Even if there was, I'd still take my chances as the ratio must be miniscule.

As for general faults, the no quibble 2 year warranty (at no extra cost) would sort out most assembly issues that may be present from new.

1 hour ago, Robse said:

This Bosch battery is one piece of crap !

Whatever.

[Cress]

We should save the  posts on this thread as an example of crowd-sourced engineering review of the S20 fire.  Really.  Our posts here outline issues in the design of the S20 that should have been addressed before Kingsong put riders on EUCs.  When manufacturing doesn't do pre-production review the public protects itself by regulating and/or banning products.  We can benefit EUC Manufacturers, the EUC community, and evolution of EUC technology by promoting technical reviews.  

I am open to admitting a mistake and I certainly could have missed something - BUT . . . It's time for EUC Manufacturers to incorporate technical reviews.  I will support effort by EUC Distributors to technically review and support EUC Manufacturers who incorporate technical product reviews.

Follow-the-money manufacturing errors are often simple but difficult to uncover.  Technical facts are understandable and every technical problem is solvable.  I care about the facts.

[RagingGrandpa]

9 hours ago, Planemo said:

My Bosch ebike battery 
I trust the pack 110%, and have less than zero concerns storing it in my house.

There are 40 Cobalt-type Li-Ion cells inside. 
I agree it may be reasonable to charge and store it indoors, but the fire risk is inherent with these cells. Never zero.
Everyone using these cells should have a conscious plan for dealing with a pack fire.
(Which door is closest? Have some heavy leather gloves? Or a tool to shove the pack out a door? Could you quickly produce a bucket of water to drown it? Extinguisher accessible nearby, to stop items in your house that it lit? etc) 

[supercurio]

Kind of off-topic but not really, I just watched the Netflix documentary "Downfall, The Case Against Boeing" and although there are many difference of course with the S20 situation, it gives a framework to interpret the statements and promises KingSong will make very soon.

Boeing, a company previously renowned for its quality engineering and safety record promised to deliver a software update within 6 weeks that would make the "MCAS" system of the 747 Max and the airplane safe, after blaming the pilots for not being able to work around the undocumented hidden defective system.

@Cress sometimes makes comparison with the aviation industry, I understand better now.

Highly recommended watch!

[Planemo]

4 hours ago, RagingGrandpa said:

I agree it may be reasonable to charge and store it indoors, but the fire risk is inherent with these cells. Never zero.

I never said the risk was zero.

The point I was making is that I trust my Bosch pack as much as I do my cellphone, which is left on charge on my bedside every night.

Similar to my HP laptop, Dyson cordless hoover, Braun toothbrush and Phillips shaver, all of which use li-ion batteries.

And FWIW, I dont subscribe to the fact that more cells (as in an EUC or ebike pack) should make me any more nervous, because even one li-ion/li-po cell going nuclear is enough to light a house up. And I don't take the lives of my family lightly.

EUC manufacturers just need to get their arse into gear. It really is that simple.

I have untold amounts of 18650's and li-po pouch packs lying around in my shed from various projects. Some have been down to 0.03v. None have gone critical, and whilst all of them need to be treated with respect, they are not the hand grenades that some seem to think they are. The issues arise when BMS's aren't up to the job or cells get shorted.

[Tawpie]

Having started my Engineering career at that company, before McDonnell Douglas bought the place using Boeing's own money and it became Chicago based Boeing, the culture shift that was driven from the top by the new management was disheartening indeed. Prior to the buyout, it was (and we said this smugly) a company "By the Engineers, Of the Engineers, For the Engineers". Afterwards, not so much... at least in the opinion of some of the Engineering troops that stayed on. Us old timers shake our heads in disbelief, and are very sad.

Anyway, if KS mahogany row (where the big bosses sit) is demanding a safe, quality, product—we'll see delays and possibly an explanation of why they're choosing whatever course of action they select. If time-to-market and the business side is driving, we need to be mindful as prospective owners and start getting used to the mindset of our more pure-performance minded brethren.

KS got a staggering number of things right in their first public 'release' of this wheel. But they got (at least) one thing (that we know of) woefully wrong. Maybe two things. The fact that so many aspects were right gives me a lot of hope that so far they've been working on designing the wheel, they haven't been throwing parts together willy nilly.

So here's to hoping that an Engineering/safety-first attitude still prevails and they suck it up, dig to the root cause and then and only then design a solution... but I know that business pressures aren't lightening up and this thing was supposed to be producing revenue quite some time ago. Master is coming. Ready or not.

[Wolverine]

14 hours ago, supercurio said:

Kind of off-topic but not really, I just watched the Netflix documentary "Downfall, The Case Against Boeing"

There was a problem mentioned in the code, but unfortunately no more words were said about it. I personally feel that this issue should have been reflected more. The fact is that Boeing's 737 Max software were outsourced to $9-an hour Indian engineers. As you can see, you can't get a working code for $9 hour tax. After the two planes crashed, it turned out that there were so many bugs and faulty lines in the code that Boeing's own IT engineers had to fix it.

More and more code is being outsourced from India and the quality is very poor. Unfortunately, no one dares to talk about it in public.

Edited April 1, 2022 by Wolverine

[Finn Bjerke]

Shouldńt Kingsong make a statement today about the S20 situation? 
Shouldnt Inmotion make a statement in January about the V13 situation? 


 

[Forwardnbak]

12 minutes ago, Finn Bjerke said:

Shouldńt Kingsong make a statement today about the S20 situation? 
Shouldnt Inmotion make a statement in January about the V13 situation? 


 

They are just hoping if they close their eyes hard enough we will all go away. 

[wstuart]

This post is a guess at what happened from a total novice.  I've only been riding for a couple months.

First I am wondering if this wheel is sensorless.  Adam (Wrongway) in his review mentiond the motor cluncking when he tried to go up steps.  I commented that I thought that the motor might be "cogging" in low rpm/high torque situations because it is sensorless.  Someone else then posted that the motor is in fact sensored, it just has the ability to operate sensorless if the sensor fails.  Is this true??

I have a background in racing rc cars.  Electric Rc racers are very familiar with the difference in performance between sensors and sensorless motors.  At low rpm/ speed when taking of or braking the motor feels like a lightswitch (not smooth) and doesn't perform well.  This condition goes away once the car is up to speed.  Also the motor can "cog" or stutter if alot is being asked of it at low speed.  

Everything I'm seeing in the S20 videos I've watched is reminding me of this.  Specifically, the s20s lackluster performance in the first 3 meters, Adam's cluncking (perhaps cogging) at low speed going up steps and finally Ustrides description what the s20 was doing just before it cut out.

 

Ustride said under extreme braking he felt the motor slip and then catch so he backed off.  He then applied brakes hard again, and the motor "slipped" and cutout.  I wondering if there are a couple things going on here.

1.  Some component of the controller is not strong enough to give the motor everything it's asking for so it momentarily skips.

2.  In these brief instantaneous moments the controller looses track of position of the commutator because it is sensorless (I believe this is what cogging is).  I think this is what Wrongway and Ustride experienced in their high torque situations. 

3.  In Ustrides case the slip of the commutator was big enough that the controller completely lost track of the position of the commutator. - I wonder if this is what the stuttering was when the wheel was laying on the ground after the fall.  The controller was trying to find the commutator and drive the motor but couldn't.  The power had no where to go and smoked the board.  I would imagine the controller and motor being in this state would put a huge draw on the batteries - almost like a short. 

4.  Some other component allowed this to damage tha batteries - like weak fuses or something.  I don't really know alot about that. 

 

Please feel free to completely shoot down my ideas here.  I'm just engaging in some amateur armchair engineering. 

[supercurio]

14 minutes ago, wstuart said:

This post is a guess at what happened from a total novice.  I've only been riding for a couple months.

First I am wondering if this wheel is sensorless.  Adam (Wrongway) in his review mentiond the motor cluncking when he tried to go up steps.  I commented that I thought that the motor might be "cogging" in low rpm/high torque situations because it is sensorless.  Someone else then posted that the motor is in fact sensored, it just has the ability to operate sensorless if the sensor fails.  Is this true??

No engineering review or reverse-engineering has been done on that, so for now we only have KingSong's statements.

14 minutes ago, wstuart said:

I have a background in racing rc cars.  Electric Rc racers are very familiar with the difference in performance between sensors and sensorless motors.  At low rpm/ speed when taking of or braking the motor feels like a lightswitch (not smooth) and doesn't perform well.  This condition goes away once the car is up to speed.  Also the motor can "cog" or stutter if alot is being asked of it at low speed.  

Interesting comparison! Hard to tell tho.

14 minutes ago, wstuart said:

Everything I'm seeing in the S20 videos I've watched is reminding me of this.  Specifically, the s20s lackluster performance in the first 3 meters,

"Lackluster performance in the first 3 meters" would be confirming tho. I'm personally not sure if it's a thing and not placebo effect.
Adam / Wrong Way said for instance than climbing ability (you know like his 40° test) was the same if starting from a standstill or a after a riding a few meters.

14 minutes ago, wstuart said:

Adam's cluncking (perhaps cogging) at low speed going up steps and finally Ustrides description what the s20 was doing just before it cut out.

Adam's clunking is associated with climbing over a step without as much jumping over it like we're used to on non-suspension wheels, if you compare to what an EXS20 does in this situation, it also dips but this video shows how bouncing of the tire and the loss of traction enters it play.

 

What @U-Stride experienced is standard over-power characteristics. Some call it magnets slipping, but it's essentially requesting more torque than the motor+controller combo can provide. Nothing special for a motor with hall sensor.

Would you say a sensorless motor/controller behavior has differences here?

14 minutes ago, wstuart said:

Ustride said under extreme braking he felt the motor slip and then catch so he backed off.  He then applied brakes hard again, and the motor "slipped" and cutout.  I wondering if there are a couple things going on here.

1.  Some component of the controller is not strong enough to give the motor everything it's asking for so it momentarily skips.

2.  In these brief instantaneous moments the controller looses track of position of the commutator because it is sensorless (I believe this is what cogging is).  I think this is what Wrongway and Ustride experienced in their high torque situations. 

3.  In Ustrides case the slip of the commutator was big enough that the controller completely lost track of the position of the commutator. - I wonder if this is what the stuttering was when the wheel was laying on the ground after the fall.  The controller was trying to find the commutator and drive the motor but couldn't.  The power had no where to go and smoked the board.  I would imagine the controller and motor being in this state would put a huge draw on the batteries - almost like a short. 

Again not sure: you can observe an identical behavior on all current wheels when over-powered.

14 minutes ago, wstuart said:

4.  Some other component allowed this to damage tha batteries - like weak fuses or something.  I don't really know alot about that. 

Please feel free to completely shoot down my ideas here.  I'm just engaging in some amateur armchair engineering. 

Heheh we all do 

[Paul A]

Is it really possible that a burning control board has caused a short circuit/spike in current draw/fire in one battery pack?

The second pack then subsequently catching alight from first pack fire?

[Tawpie]

11 minutes ago, Paul A said:

Is it really possible that a burning control board has caused a short circuit/spike in current draw/fire in one battery pack?

The second pack then subsequently catching alight from first pack fire?

Yes, but the first cascade should NOT happen (it still might, but it also still shouldn't). A pack sitting on top of a fire also catching on fire is totally expected.

[Paul A]

Thanks.

If U-stride is willing to be a guinea pig again, perhaps should repeat the event on another S20.

Logged on and video to capture data.

If fire results again, would eliminate prior crash as a factor.

[supercurio]

13 minutes ago, Paul A said:

Is it really possible that a burning control board has caused a short circuit/spike in current draw/fire in one battery pack?

Yes but only with a BMS and wiring capable of handling currents so high that cells get in the danger zone.

And so far it has not been the case,. For instance, @RagingGrandpa destructive testing showed that the previous Gotway's packs had a point of failure in the BMS acting as fuse, either intentionally or by design if the packs were shorted, before cells would overheat.
Likewise, plenty of Sherman mainboard went in flames shorting the packs at the same time, but that hasn't even been enough to blow the 2x30A fuses.

In order to see such short turn into a fire, both the BMS hardware need to handle significantly more current than we've seen before and its fuse must have been ineffective (not unlikely: tiny 60V rated SMD fuse)
I'm not even talking about the MOS on the BMS board since we don't know their use. Maybe they also blew due to the regen voltage spike and contributed to the fire escalation.

It is unclear so far what the influence of the BMS fuse had on the cells nearby, probably arcing the whole time the board was burning, but we can guess that it generally didn't help.

13 minutes ago, Paul A said:

The second pack then subsequently catching alight from first pack fire?

Yes it looks like we can see that in the video.
I thought top back then bottom pack but @Tawpie wrote the opposite earlier.

[mrelwood]

 

4 hours ago, wstuart said:

Someone else then posted that the motor is in fact sensored, it just has the ability to operate sensorless if the sensor fails.  Is this true??

Yes, in that sense the motor itself is an ordinary BLDC motor with hall sensors. The ability to keep running in case a sensor fail is a controller feature.

 

4 hours ago, wstuart said:

Electric Rc racers are very familiar with the difference in performance between sensors and sensorless motors.

AFAIK RC cars don’t have BLDC motors, so the behavior is different than on an EUC. While some features or issues can seem familiar, they can come from completely different causes.

 

4 hours ago, wstuart said:

Specifically, the s20s lackluster performance in the first 3 meters, Adam's cluncking (perhaps cogging) at low speed going up steps and finally Ustrides description what the s20 was doing just before it cut out.

The lackluster performance at the start is a designed firmware feature to avoid large current spikes. Adam’s clunking came from the motor trying to catch up after being overleaned at a small speed, and Ustride’s soft balancing behavior was probably one of the large caps that had already failed.

 

4 hours ago, wstuart said:

1.  Some component of the controller is not strong enough to give the motor everything it's asking for so it momentarily skips.

An EUC motor is a dummy component, and it doesn’t ”ask” for anything. The controller drives the motor with as much power as the controller itself deems necessary for balancing. But when overleaned, like you wrote the controller is not able to provide the power required to keep the wheel upright.

 

4 hours ago, wstuart said:

2.  In these brief instantaneous moments the controller looses track of position of the commutator because it is sensorless

It is not. As far as we know, the hall sensors of the S20 didn’t fail as nothing clearly points to that.

 

4 hours ago, wstuart said:

I wonder if this is what the stuttering was when the wheel was laying on the ground after the fall.

The stuttering was probably caps and mosfets of the board and even the first battery cells exploding as the controller had already completely failed. Some mosfets shorting could also cause a twitch.

 

4 hours ago, wstuart said:

The power had no where to go and smoked the board.

When the power has nowhere to go, it stays in the batteries. The full power of the battery is always present at the controller, and the mosfets are used to drive some, all, or none of the power to the motor. In this case the board most probably failed in a way that it shorted the battery.

 

3 minutes ago, Paul A said:

Is it really possible that a burning control board has caused a short circuit/spike in current draw/fire in one battery pack?

Yes. When a mosfet blows, it can create either a direct shirt for the battery, or remain in an open state providing almost a short to the battery via the motor coils. For the battery to fail that fast, the mosfets may have caused a direct sort.

3 minutes ago, Paul A said:

The second pack then subsequently catching alight from first pack fire?

Yes, it was essentially thrown into a bonfire, so it joined the party.

[Tawpie]

The report is up https://www.kingsong.com/kingsongnotifications.html#images-1

[Tawpie]

So... two levels of failure. They don't have the motherboard yet so they're guessing it started with the cutoff, and don't know why but mention a possible hall sensor failure that caused a MOSFET failure but they don't provide a guess as to why.

Regardless of "why", the battery shorted and the next two layers of protection failed. They believe a SW error in the protection cutoff control reset before the fuse could blow—it shut things down, but restarted with the battery connected. Then shut down again and repeated the cycle. But the short didn't exist long enough for the fuse to blow even though the nickel strips were red hot... and fire results.

FW changes to make a short circuit/overcurrent software detection persistent through power cycles, reset by connection to charger or disconnecting the BMS from the load (controller+battery). Boot into short circuit protection mode immediately. Other changes (power increase, balancing etc.)

OPINION

Elated to see a detailed report—this is landmark in our little corner of the world. Glad they were able to physically reproduce the red hot nickel strips. The chosen SW changes seem "advisable" in light of what they were able to reproduce.

Also happy to see they didn't spend much time on why the battery got shorted and have concentrated on the failure of the BMS to protect the battery. I'm happy because they don't really have information about the start of the incident and would be speculating (as we do, but it's all we can do). I'm happy they're augmenting their logging function to try to get more data.

I'm back to being every excited for June. Or July. I'm torn about having them put in a fuse that's easier to blow... don't want to have a false-positive take out the fuse and require a battery swap as added insult to a faceplant.

Edited April 2, 2022 by Tawpie

[BatteryMooch]

Thank you @Tawpie for posting that!

Wow, multiple software and hardware failures at work here. IMO all of these issues should have been found early in the bench testing. Especially with them being so easy to identify now.

Some thoughts…

• Intermittent/repeating short circuits like these should be expected (eventually, though rarely) in a high vibration environment and the BMS/ESC/pack tested to make sure these types of short circuits are not a problem.
• The fuse not blowing is huge…HUGE. This component must be properly spec’d to prevent exactly this kind of unexpected fire hazard. 
• Increasing the thickness of the nickel strip they use would make pack assembly a bit tougher, and a touch more expensive, but would give the fuse more time to blow before the heat from the nickel strip forced some cells in the pack to go into thermal runaway (catch fire).
• They didn’t mention any hardware changes/fixes though they might be waiting for the pack/BMS analysis to be done. IMO they won’t be able to say anything definitively though about the cause after this analysis due to the damage. Best to just design a much more robust system than they have now and actually TEST it.

Still digesting what their report said though and reading the other technical responses about this.…great bunch of knowledgeable engineers here! Looking forward to KS sorting it all out though and a more robust S22 coming to market.

Edited April 2, 2022 by BatteryMooch