Lilvodka Posted June 23, 2019 Share Posted June 23, 2019 6 hours ago, Jason McNeil said: Apologies for the late attendance to the party. Since receiving a 2nd board failure (this case was different, it was powered on from a stand still & didn't respond), including Marty's, within two weeks of shipping out the first batch of Nikolas, I've reached out to Linnea on Thursday to try to come up with an action plan to see what options are available to greatly reduce/eliminate the risk of these MOSFET thermal overload events. The evidence from both of these failures has marked similarity in outcome (see below), where this glue may be acting as a thermal barrier, not helped by the absence of thermal paste on the underside of the metal heat transfer surface. One option being explored, is to recall the boards. remove this glue, & apply the thermal paste. To date Gotway have shipped 600 units; they say they have not received reports from other regions of this failure—it's likely more will trickle in in time. Putting out an official recall is not something to be taken lightly; in their defense, they're probably looking for a few other reports, before taking this step. Gotway have a lot of experience with building high power motor vehicles; they had gone through several rough patches, particularly with reliability of earlier generation boards, with the weaker MOSFETS, but at least then they were pretty consistent with applying liberal quantities of thermal paste to this underside. It was baffling to me why they would go back to an inferior TO22x FET package AND now we learn about the lack of thermal paste. From my experience, such oversights are usually not the result of penny-pinching, but execution on the production line, where the factory needs a ruthless eagle-eyed floor manager, to see that every step of assembly is done by the book. The silver-lining to all this, is that an early finding in the production cycle is a necessary facet to a small-scale non-ISO production facility, where the manufacturer's financial pain of having to deal with a crisis, means that they will be imbued with a sense that this situation must never occur again. For all current Nikola owners, please hang in there, we should have an action plan in the next few days. EDIT: To anticipate questions on the 100V Nikolas, they are using the TO247 MOSFET package. The first production of these special edition 1845Wh variants is still underway, so there is assurance that in light of this episode, they will getting the thermal paste treatment! Do you know if this issue extends to the newest Tesla (speaker and anti-lift version)? 1 Quote Link to comment Share on other sites More sharing options...
fbhb Posted June 23, 2019 Share Posted June 23, 2019 (edited) 1 hour ago, Topkek said: Do you know if this issue extends to the newest Tesla (speaker and anti-lift version)? At this stage I don't believe there have been any reports of this issue with the latest Tesla board. The board used for the Tesla V2, was already an upgraded board that actually began getting installed in the Tesla V1 quite some time ago and was distinguished by a change to the "ringlight" pattern options (sadly losing the brake light option) so it has been around for a decent period of time now. However, as mentioned elsewhere and possibly of some concern, the construction method is the same as the Nikola having one row of hidden mosfets along the centre of the board (so who knows whether the hot glue assembly method may be a factor here too!) Edited June 23, 2019 by fbhb 1 2 Quote Link to comment Share on other sites More sharing options...
Popular Post Harold Farrenkopf Posted June 23, 2019 Popular Post Share Posted June 23, 2019 @Jason McNeil, no thermal paste when using the thermal conductive pad. When the thermal paste is used where no electrical isolation is required, only the minimal amount is required to fill any air gaps in the irregular surfaces even though they look flat and smooth. One simple way of placing the pad in place with the spacers for the screws is to use a made tool for the task that looks like a 6 pronged comb that goes through the holes of the pc board and the mosfet holes and the pad slid on it and then the comb put into the heatsink holes and then slid down to the board. Then place the screws in place with no need for any glue or paste to hole the pad aligned. And if they really want a easy and fast precision assembly, use an assembly that has 2 rows of 6 to assemble all mosfets at once. 4 4 Quote Link to comment Share on other sites More sharing options...
Planemo Posted June 23, 2019 Share Posted June 23, 2019 Or they could just use an easily de-mountable piggyback motherboard like the Z10 so that all the FETs can be reached and affixed easily without having to juggle around holding things in place with poor access. Quote Link to comment Share on other sites More sharing options...
Marty Backe Posted June 23, 2019 Author Share Posted June 23, 2019 1 hour ago, Harold Farrenkopf said: @Jason McNeil, no thermal paste when using the thermal conductive pad. When the thermal paste is used where no electrical isolation is required, only the minimal amount is required to fill any air gaps in the irregular surfaces even though they look flat and smooth. One simple way of placing the pad in place with the spacers for the screws is to use a made tool for the task that looks like a 6 pronged comb that goes through the holes of the pc board and the mosfet holes and the pad slid on it and then the comb put into the heatsink holes and then slid down to the board. Then place the screws in place with no need for any glue or paste to hole the pad aligned. And if they really want a easy and fast precision assembly, use an assembly that has 2 rows of 6 to assemble all mosfets at once. Exactly. It took me about 60-seconds to think of this. Why the Chinese can't is beyond me. Quote Link to comment Share on other sites More sharing options...
Popular Post Jason McNeil Posted June 23, 2019 Popular Post Share Posted June 23, 2019 (edited) Another point I forgot to mention last night: when the first tear-down photos were posted by EcoDrift, I wrote a list of questions about the Nikola, where the selection of these MOSFETs was at the top of this list. MOSFETs: What type of MOSFETs are used in the Nikola, are they HY brand, what model? Why are they not the TO-247 type found in the MSX? Is there now max current regulation in the firmware? If so, what is this value, 100A max? What sets Gotway apart from other Manufacturers, is that their firmware (as of a year ago) does not perform current regulation—this is why their controllers do not contain fuses. I lobbied pretty hard for the TO247 on the Nikola, but the answer I got was that the Wheel had been extensively tested, that fears of going back to bad old days of controller blowouts were unwarranted... 7 hours ago, Unventor said: Having a dealer like Jason to breach the culture differences and taking discussions with manufacturers like he does is simply but nothing less than priceless. You give me too much credit here. Gotway Engineers are no fools, when contrasted to an organization like Ninebot with their vast resources, Gotway have usually been open to reason, & back up their products. 2 hours ago, Harold Farrenkopf said: no thermal paste when using the thermal conductive pad. When the thermal paste is used where no electrical isolation is required, only the minimal amount is required to fill any air gaps in the irregular surfaces even though they look flat and smooth. This may be true, we need to commission a test rig for stressing the controllers under various loads in test these principles experimentally. I could be wrong, but I suspect that with a drop of efficient thermal compound on the conductive metal surface, will make an dramatic effect on shuttling away heat. In most applications these components are probably not stressed to anywhere to the same degree with as on a Wheel charging up a hill on a hot day, where the standard insulating pad is adequate. Also a special thanks to @Marty Backe & @Phil McLaughlin for their efforts with this investigation. Edited June 23, 2019 by Jason McNeil 1 3 Quote Link to comment Share on other sites More sharing options...
Popular Post Simone Malicius Posted June 23, 2019 Popular Post Share Posted June 23, 2019 11 hours ago, Jason McNeil said: Apologies for the late attendance to the party. Since receiving a 2nd board failure (this case was different, it was powered on from a stand still & didn't respond), including Marty's, within two weeks of shipping out the first batch of Nikolas, I've reached out to Linnea on Thursday to try to come up with an action plan to see what options are available to greatly reduce/eliminate the risk of these MOSFET thermal overload events. The evidence from both of these failures has marked similarity in outcome (see below), where this glue may be acting as a thermal barrier, not helped by the absence of thermal paste on the underside of the metal heat transfer surface. One option being explored, is to recall the boards. remove this glue, & apply the thermal paste. To date Gotway have shipped 600 units; they say they have not received reports from other regions of this failure—it's likely more will trickle in in time. Putting out an official recall is not something to be taken lightly; in their defense, they're probably looking for a few other reports, before taking this step. Gotway have a lot of experience with building high power motor vehicles; they had gone through several rough patches, particularly with reliability of earlier generation boards, with the weaker MOSFETS, but at least then they were pretty consistent with applying liberal quantities of thermal paste to this underside. It was baffling to me why they would go back to an inferior TO22x FET package AND now we learn about the lack of thermal paste. From my experience, such oversights are usually not the result of penny-pinching, but execution on the production line, where the factory needs a ruthless eagle-eyed floor manager, to see that every step of assembly is done by the book. The silver-lining to all this, is that an early finding in the production cycle is a necessary facet to a small-scale non-ISO production facility, where the manufacturer's financial pain of having to deal with a crisis, means that they will be imbued with a sense that this situation must never occur again. For all current Nikola owners, please hang in there, we should have an action plan in the next few days. EDIT: To anticipate questions on the 100V Nikolas, they are using the TO247 MOSFET package. The first production of these special edition 1845Wh variants is still underway, so there is assurance that in light of this episode, they will getting the thermal paste treatment! @Jason McNeil let me honestly tell you something.. when I started to get curious about euc (6 months ago.. I'm from Italy) your name was like a form of insurance in all sort of post/feedback/web search I made, whenever I've read "Jason" I felt like I was on the right path to know more about euc! then I found this forum and I knew why your name was a sign of guarantee! sorry for this ot but you have to know how important is what you do in the way you do it! 5 3 Quote Link to comment Share on other sites More sharing options...
LucasD Posted June 23, 2019 Share Posted June 23, 2019 There is of course chance that new guy come to line and used hot glue to help him. As we don't know how many guys assemble and how many uses hot gun, it is really hard to judge. But given that we know of 2 cases, gateway need to investigate it... 1 Quote Link to comment Share on other sites More sharing options...
Popular Post Jon Stern Posted June 23, 2019 Popular Post Share Posted June 23, 2019 23 minutes ago, LucasD said: There is of course chance that new guy come to line and used hot glue to help him. As we don't know how many guys assemble and how many uses hot gun, it is really hard to judge. But given that we know of 2 cases, gateway need to investigate it... Well that wouldn't be a good sign either. In manufacturing it's vital that each operator is trained on, and follows, the Standard Operating Procedure (SOP). This should be a document that provides precise, step by step instructions on how to perform the build. There is no room for interpretation. It's the job of QC to make sure that the SOP is followed. If there's an operator who is deviating from it, they should be taken off the line and retrained. If the QC doesn't catch these things, that a huge red flag too. What else are they missing? 3 2 Quote Link to comment Share on other sites More sharing options...
Popular Post Chriull Posted June 23, 2019 Popular Post Share Posted June 23, 2019 (edited) 3 hours ago, Harold Farrenkopf said: @Jason McNeil, no thermal paste when using the thermal conductive pad. When the thermal paste is used where no electrical isolation is required, only the minimal amount is required to fill any air gaps in the irregular surfaces even though they look flat and smooth. +1. 1 hour ago, Jason McNeil said: .... I could be wrong, but I suspect that with a drop of efficient thermal compound on the conductive metal surface, will make an dramatic effect on shuttling away heat. In most applications these components are probably not stressed to anywhere to the same degree with as on a Wheel charging up a hill on a hot day, where the standard insulating pad is adequate. Thermal grease is just used to fill the small irregularities, as the contained air is a great thermal isolator. But still thermal grease, pads, etc are much worse thermal conductors than the metal of the heatsink and the mosfet - so as less as possible should be used. Just enough to fill the air gap. In the case of the EUC heatsinks with all MOSFETs mounted electrical isolation is needed, too. This could be achieved with thin mica or other ceramic plates and a thermal grease layer and _both_ sides. Most probably an assembly "nightmare"! They used thermal pad (should) fullfill both tasks - fill up the air gaps, electricly insulate and still provide sufficient thermal conductivity. Additional application of thermal grease should worsen thermal conductivity?! To optimize the quality one could do a market research for different thermal pads - there is quite a range of different qualities available. (as is true for thermal grease) A second point/thought to improve/ensure thermal conductivity could be using clips instead of screws to mount the mosfets on the heatsink plate: the silicium die, where the heat is generated is more or less in the middle of the mosfet body, assembled on the back metal plate of the mosfet. This point needs the pressure for good contact to the heatsink. (Afaik thermal pads need to be "correctly" compressed to perform as specified in the datasheet). As the screwhole is "far" away from the die there could be a leverage effect bending the body with the die a bit away from the heatsink, especially if too much torque is applied. Edited June 23, 2019 by Chriull 5 Quote Link to comment Share on other sites More sharing options...
Chriull Posted June 23, 2019 Share Posted June 23, 2019 1 hour ago, Jason McNeil said: Is there now max current regulation in the firmware? If so, what is this value, 100A max? Afair some times ago (~?msuper v3?) the official statement from the GW representative was that the firmware limits the current to 120A. Imo this can only be survived for short power peaks - such currents leads to dissipated powers in the range of ~140W (~120A² * 10mOhm), with paralleled mosfets to a fourth of this - but with no real guaranteed of equal distribution. So measuring the compartment temperature instead of the heatsink temperatures near the mosfets can easily lead to too much time delay, overheating and destroying the mosfets. 1 Quote Link to comment Share on other sites More sharing options...
Popular Post Phil McLaughlin Posted June 23, 2019 Popular Post Share Posted June 23, 2019 52 minutes ago, LucasD said: There is of course chance that new guy come to line and used hot glue to help him. As we don't know how many guys assemble and how many uses hot gun, it is really hard to judge. But given that we know of 2 cases, gateway need to investigate it... Excellent point! Yes, we do not know whether all of the Nikola assemblers are using the glue trick to help attach the heatsink. I just wanted to add that we now have photographic evidence of THREE boards that have contamination that we believe is hot glue on the MOSFET tabs. The pictures shared by @Marty Backe and @Jason McNeil were examples of blown boards with this anomaly. My Nikola board had not failed. I chose to strip it down for inspection/rebuild as a precaution after seeing the pictures of Marty's board and having the feeling that this "glue" problem was not isolated to his Nikola. My board that showed clear evidence of contamination but no visible damage to the MOSFETs demonstrates that boards assembled this way are not guaranteed to fail within 100 miles or so of ordinary use. This does not necessarily mean that they will be safe or reliable in the long term if this assembly technique was used. Personally I chose not to ride this wheel until I corrected the heat transfer situation but I fully respect that others may judge the risk differently. I truly hope that I was overly cautious and my teardown/rebuild will turn out to have been unnecessary. 6 3 Quote Link to comment Share on other sites More sharing options...
scotchtape Posted June 23, 2019 Share Posted June 23, 2019 So every gotway wheel had this issue? Including msx? Quote Link to comment Share on other sites More sharing options...
LucasD Posted June 23, 2019 Share Posted June 23, 2019 I would say it depends on board complexity and individual assembler. Also to give some peace of mind, remember this hot glue is still better than air, but worse than thermal pad/ paste. Air> hot glue>thermal pad>thermal paste > toothpaste (for short time, as overclockers knows ) Quote Link to comment Share on other sites More sharing options...
meepmeepmayer Posted June 23, 2019 Share Posted June 23, 2019 (edited) 2 hours ago, Jason McNeil said: MOSFETs: What type of MOSFETs are used in the Nikola, are they HY brand, what model? Why are they not the TO-247 type found in the MSX? Is there now max current regulation in the firmware? If so, what is this value, 100A max? Did he answer? If not: The 84V Nikola (for now) uses HY3712P mosfets in a TO-220 package. Source: EcoDrift in the comments of this video. If you look at the data sheet, you see 170A continuous/585A(?) max. For comparison, the MSX has mosfets (I don't know which) with around 200A cont/600A max. So the mosfet change is no big downgrade itself. But I guess the smaller package instead of TO-247 will have a huge influence on cooling. But given that wheels like the Tesla have survived with smaller mosfets than the Nikola (right?), I'm not convinced the mosfets itself are a problem. But of course it would be better if they just switched to the MSX setup for the 84V Nikola, which is apparently pretty much what they have done for the 100V Nikola. So if you could tell them that (again) Edited June 23, 2019 by meepmeepmayer 3 Quote Link to comment Share on other sites More sharing options...
meepmeepmayer Posted June 23, 2019 Share Posted June 23, 2019 51 minutes ago, scotchtape said: So every gotway wheel had this issue? Including msx? We don't know. Gotway need to figure out how widespread this problem is. But there haven't been any failure reports from other 12-mosfet wheels like the MSX or Monster (the upgrade with the MSX board). So either the problem isn't there, or the stronger mosfets survived despite glue. Either way, no need to worry about existing non-Nikolas for now The MSX board has proven to be exceptionally strong and problem-free. Just look at @EUC GUY's car push test - the cabling died before the mosfets! So even if there is glue there, it's still no problem. But of course you don't want glue there, even if it does nothing. 1 1 Quote Link to comment Share on other sites More sharing options...
Popular Post mrelwood Posted June 23, 2019 Popular Post Share Posted June 23, 2019 1 hour ago, scotchtape said: So every gotway wheel had this issue? Including msx? Probably not, but I will still dismantle mine and check as soon as I get to it. The larger TO-247 package has the screw hole in the middle of the Mosfet, which makes the thermal connection a lot more solid against pulling/twisting the Mosfet legs, and overtightening. If glue is found in the MSX as well, this could explain why there haven’t been issues. 1 3 Quote Link to comment Share on other sites More sharing options...
Popular Post esaj Posted June 23, 2019 Popular Post Share Posted June 23, 2019 1 hour ago, meepmeepmayer said: Did he answer? If not: The 84V Nikola (for now) uses HY3712P mosfets in a TO-220 package. Source: EcoDrift in the comments of this video. If you look at the data sheet, you see 170A continuous/585A(?) max. For comparison, the MSX has mosfets (I don't know which) with around 200A cont/600A max. So the mosfet change is no big downgrade itself. But I guess the smaller package instead of TO-247 will have a huge influence on cooling. Don't give too much value to the theoretical maximums of the datasheet. They're kinda like the range-values of wheels, you can use them to compare one mosfet (wheel) to another, but in real-life, other factors limit you to much lower numbers (at least in continuous current / power dissipation). TO-247 can take more abuse thanks to a much larger contact area for heat dissipation (at least when properly connected to a heatsink or whatever for dissipating the heat). The more important numbers are the Rds(on) (internal resistance when fully conducting) and the thermal resistances (which are usually the same for the same casing), but you'd need to know the average current / power dissipation and heatsink thermal resistance to ambient to calculate anything more meaningful. 3 5 Quote Link to comment Share on other sites More sharing options...
Marty Backe Posted June 23, 2019 Author Share Posted June 23, 2019 6 hours ago, Jon Stern said: Well that wouldn't be a good sign either. In manufacturing it's vital that each operator is trained on, and follows, the Standard Operating Procedure (SOP). This should be a document that provides precise, step by step instructions on how to perform the build. There is no room for interpretation. It's the job of QC to make sure that the SOP is followed. If there's an operator who is deviating from it, they should be taken off the line and retrained. If the QC doesn't catch these things, that a huge red flag too. What else are they missing? Jon, I admire your rattling off of standards, etc., in multiple posts, but these are Ma & Pop Chinese companies Probably at least a couple of years ago I gave up any allusion to believing that they would ever operate or behave like Western companies. 2 1 Quote Link to comment Share on other sites More sharing options...
TuN3M@N Posted June 23, 2019 Share Posted June 23, 2019 (edited) 4 hours ago, meepmeepmayer said: ... So the mosfet change is no big downgrade itself. But I guess the smaller package instead of TO-247 will have a huge influence on cooling. The troll again :D... As I said befor 7w for 1minute... Maybe this wasn't so wrong ... As Jason said, the second was not burned on a long steep hill. MSX has an open shell... This could be named "cooling". Edited June 23, 2019 by TuN3M@N Quote Link to comment Share on other sites More sharing options...
Popular Post Jon Stern Posted June 23, 2019 Popular Post Share Posted June 23, 2019 (edited) 32 minutes ago, Marty Backe said: Jon, I admire your rattling off of standards, etc., in multiple posts, but these are Ma & Pop Chinese companies Probably at least a couple of years ago I gave up any allusion to believing that they would ever operate or behave like Western companies. Oh, believe me, I'm under no illusions when it comes to expectations for manufacturers like Gotway and King Song. Even large Chinese contract manufacturers that I've worked with have to be pushed and constantly monitored to stick to the processes and apply effective quality controls. What I'm pointing out are the best methods we know of to develop and manufacturer consumer products that have low chances of failing at unacceptable rates, and in the case of a vehicle will be predisposed to being safe. While we continue to buy for these cowboy operations, we should not expect first class outcomes. We should not be surprised when our wheels throw us under an 18-wheeler! The situation isn't going to change over night, or even in two years, but continued consumer pressure can make a difference. Once these companies start to realise that there is value in their brand that needs protecting, they will begin to look at ways to minimize the risk of damaging their brand with each new product release. Edited June 23, 2019 by Jon Stern 6 3 Quote Link to comment Share on other sites More sharing options...
eddiemoy Posted June 24, 2019 Share Posted June 24, 2019 On 6/20/2019 at 11:56 PM, Marty Backe said: The Great Hot-Glue Fiasco OK folks, here's the new theory. The "stuff" that was stuck to my burned MOSFETs was thought to be a thin plastic backing from the thermal transfer material. But it never quite seemed like plastic film to me, or melted plastic film. A bit rubbery actually as I tried to peel it off. I believe it's hot glue. It melts easily when I touch it with my hot-glue gun. So I did a little experiment. I put a dollop of hot glue on one of the MOSFETs and pressed the thermal transfer sheet onto. A little bit later I pulled it off and it peeled off just like the other one. Here are some pictures that show the sequence of my experiment Not a perfect match, but who knows what type of hot-glue or applicator is being used. After poking and melting the original substance, I can say it's some type of hot-glue. Why? To assemble the circuit board to the heat-sink plate, the board has to be flipped over and the thermal transfer sheet must be between the MOSFETs and the heat-sink. After flipping the board, it's easy for the assembler to lift up the board a little and slide the sheet under the MOSFETs. But the inner row of MOSFETs (the one with the blown MOSFETs shown above) can't be easily accessed to slide the thermal sheet under the MOSFETs. Imagine yourself trying to do this. So instead of developing a fixture to align everything properly, I'm theorizing that the assembly line folks pulled out the glue-gun (the same one that they use to stick the pedal magnets on the shell and a bunch of other components - us Gotway owners know that Gotway loves hot-glue) and glued the thermal sheet to the MOSFETs so the sheet would stay in place when they flipped the board. Seems to make sense to me, knowing Gotway and their tight oversight of QC . Still waiting for feedback from EWheels. I'm pretty sure Jason is in communication with Gotway so hopefully we'll learn the real scoop eventually. But if my theory holds, seems to me that every Gotway wheel that has an inner row of MOSFETs (not just the Nikola) is a time-bomb waiting to blow. Now I think the other wheels (MSX, Monster, etc) are using the bigger MOSFETs and the hot-glue thermal barrier might not be an issue for those. But when they switched to the smaller MOSFETs, the assembly process became the weak link. BTW, hot-glue melts at ~120-degrees Celsius. That's probably hotter than these MOSFETs are supposed to get, so the hot-glue likely stays a solid and acts as a nice thermal break. This is my theory, and I'm sticking to it I miss my Nikola Can't be hot glue. First of all what hot glue do you know looks like a perfect thin sheet of plastic. Any thin sheet of plastic would melt with heating tip. If it happen like theorized, then it would be globs, not a perfectly flat sheet inbetween the mosfets. Quote Link to comment Share on other sites More sharing options...
Planemo Posted June 24, 2019 Share Posted June 24, 2019 I think you're looking at the wrong bit... Quote Link to comment Share on other sites More sharing options...
Marty Backe Posted June 24, 2019 Author Share Posted June 24, 2019 4 hours ago, eddiemoy said: Can't be hot glue. First of all what hot glue do you know looks like a perfect thin sheet of plastic. Any thin sheet of plastic would melt with heating tip. If it happen like theorized, then it would be globs, not a perfectly flat sheet inbetween the mosfets. Did you see Jason's picture of the other failure? Looks like a sloppier application of hot glue, to me anyway. I guess you'd have to see it up close and touch it. Feels and acts just like hot glue. Quote Link to comment Share on other sites More sharing options...
stephen Posted June 24, 2019 Share Posted June 24, 2019 14 minutes ago, Marty Backe said: Did you see Jason's picture of the other failure? Looks like a sloppier application of hot glue, to me anyway. I guess you'd have to see it up close and touch it. Feels and acts just like hot glue. Nearly all the mosfets are burnt and some are incinerated 😬😬 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.