So I may have killed my Mark 2 and I am not sure why or how to fix it. After finally getting documentation on Dinkum, I decided to try using a different image as described in the documents. I flashed a new thumb drive and shut down the Mark 2. However, after shutting down the Mark 2, the fan kept running for 15 minutes even after the screen was blank. I had accidentally unplugged the Mark 2 before while it was on and it had booted back up, so I didn’t think anything off cutting the power at this point.
Then I tried removing the old flash drive with Dinkum and putting in a new one before reconnecting the power. The fans came on full blast again, but the screen remained black. Deciding I must not have done something right after waiting 30 minutes, I unplugged it again since I didn’t have any other way to shut it off before putting the Dinkum flash drive back in. Then I switch back to the Dinkum flash drive, but it’s doing the same thing where the fan is on at full blast for 30 minutes and nothing is happening on screen.
Is there an SD card that might be used in the booting process that possibly got corrupted? Is it possible the flash drive itself got corrupted?
I’m at a total loss. I’d like to repair it but don’t know where to start.
I am in the same boat. I ran into an issue where my Mark II wouldn’t show the SSH option on the Devices page. So after rebooting a couple times, I tried to re-flash the Dinkum image on the USB that came with my Mark II. Aaaaaand, I had the same result as you with the fan on and blank screen.
Has anyone tried and successfully re-image their Mark II?
Is there a link somewhere to the Dinkum image? I did make a backup of my USB before starting to tinker, but it would be nice to know there’s a clean/factory image of what shipped with the Mark II that I could restore in a pinch.
I just downloaded a fresh copy to verify that we didn’t accidentally push a bad image to the CDN, and that is booting fine.
It’s possible that your display cable has come loose. We discovered this on a small number of early units and have made changes to our assembly process to mitigate this. If you’re comfortable doing so, you can open the front of the Mark II with a philips head screw driver and check the display DSI ribbon cable by giving it a gently tug. Some more detailed instructions for that here:
Hi all, if anyone runs into this same issue - I’m very keen to know the filenames of all images you’ve booted on the device. Particularly if you tried using the old Dev Kit Stable image which would be named: MarkII_arm-rpi64_Prod_013-stable_2021-12-13.img.gz
NOTE: Do not try to use this image in a production Mark II. It may be the cause of this issue, and even if it’s not - it won’t work on the newer hardware. All the new software however works on all previous iterations of the hardware.
For at least one device in this state it is the Raspberry Pi itself that is borked. Eg that Pi will no longer boot anything at all, even in isolation of the rest of the Mark II.
It might be just a bad Pi but given there are a couple of people reporting similar behaviour and they all seem to have booted fine the first time, and then failed after reflashing attempts, it seems far more likely that something is causing the failures.
Hi - My Mark II has been running since early Nov 2022. It was fine yesterday morning. Then yesterday afternoon, I returned after a few hours away and found the screen blank and the fan running loudly. I tried powering it off, waiting some time, and powering it back on. The screen is still blank, and the fan is running loudly. Pressing buttons on the top of the box does not illuminate any LEDs. It does not respond to ping on the IP address to which I had connected in the past. The “My Mycroft” portal indicates that the device has been disconnected for 20 hours. How can I go about debugging this? I have been using whatever flash image was shipped with the Mark II; I have not tried to update it myself. Many thanks! -Bill
My Mark II is showing the exact behavior as @ashmanskas. It was working this morning, screen went blank with fan running full-speed. Rebooting does not resolve it. I do see red glow inside:
I’m having the same issue as well. I’ve been troubleshooting and found that the included thumb drive is dead (to the point where I couldn’t even create a backup iso). I flashed an iso to a spare thumb drive and now it’s working again.
My guess is that the flash is failing in these thumb drives.
Many thanks for the tip. I also replaced the flash drive, with the “stable” image downloaded from mycroft-software, and my Mark II is again up and running (though I did need to redo the configuration).
I also observed that the very small SanDisk flash drives were getting quite hot. It’s one reason we went with a different drive when we purchased some to load our Neon OS on. They’re still small, but they’re metal on the outside, so they dissipate the heat better.
For those of you ordering a new drive, I suggest upgrading to an SSD. We’ve gotten faster performance of our Neon OS with one. It’s only a little edge for most of the on-device skills, but it boots faster and runs noticeably faster for skills that access external data sources. I’d be interested in hearing anyone’s experience who tries using an SSD with the Dinkum or OVOS OS.
On the flip side, I don’t recommend using a random old USB drive from the back of a desk drawer - I loaded our OS on one, a USB 2.0, and it was noticeably slowed.
Another troubleshooting suggestion is to try an alternate OS.
One user told me that he thought his issues were hardware, but when he tried booting his Mark II with our OS the issues were absent, so he was able to realize it was a software issue, not hardware.
If you are experiencing this boot failure, one option would be to test your Mark II with the alternate Mark II operating system “Neon OS”.
You can flash your own USB with the Neon OS that can be downloaded here at no charge: Neon AI OS for the Mark II | Neon AI
The Neon OS is a fork of the Mycroft OS, and is also open source software.
You can also order the Neon OS on a new 64GB USB 3 drive on Square for $19.95 - please note this is not a sales pitch since we don’t make money on this. Neongecko is a Mycroft AI Channel Partner and Mycroft Community Developer.
While its possible that the OS could be at play, I personally observed that the data storage itself was damaged; I wasn’t even able to create a backup iso of the drive. If it was an issue with data on the filesystem, i would have been able to create an image containing the corrupted data.
In general, flash memory has a relatively low number of read/write operations before it no longer can be used. Since thumb drives are more geared to infrequent writing of files, they lack the features SSD and more expensive sd cards possess. Its also worth adding that one such feature, TRIM/fstrim, requires support from both the hardware and software. I haven’t dug in to the mycroft iso, but its an area which could potentially be optimized in future builds.
@devnill brought up an excellent point about flash memory having a limited lifespan. That’s something I’ve never had to consider before. I’m just starting down the research path on that, keywords are endurance & P/E cycles. So far, it looks like high temperature reduces the lifespan of flash memory, so switching to the small but metal USB we chose, an SSD, or if it won’t be awkward for you a larger metal USB should give it a longer lifespan.
If you find a choice that looks better than ours, and any more pertinent information, please do share that as well.
If anyone wants to try an SSD, we shopped around a bit for a good value, and chose this one from among many good choices. I tested and am happy with it, so we set it up as an option on Square also. We’re not an Amazon affiliate.
mine is garbage and I can’t believe I paid money for this. it only boots sometimes and I had to buy it direct because I probably won’t ever see my kickstarter one.
holy crap I should’ve made my own with a raspberry pi.
