Timing out to api.mycroft.ai - is there a problem at mycroft.ai?

Hi,
I am a complete noob here, so sorry if this all seems stupid. - I performed two clean installs on OpenBox VM’s yesterday ( one LinuxMint and one on Lubuntu) and got them both working ( after a bit of burning of sulpher and sacrificing black chickens, of course :slight_smile: )

Anyway, they are working OK, BUT I am finding now that both of them are showing an error in the CLI " ReadTimeout: HTTPSConnectionPool(host=@api.mycroft.ai’, port=443): Read timed out. (read timeout=15) "

Is that normal?
The reason I ask is that when I make a change to my “home.mycroft.ai” skills, it is not reflected on the client - for example, I added “podcast-skill” yesterday and added an RSS feed on home. , and it worked & still works, but when i try to make any changed there tonight the webpage says " Sorry our servers are down right now and will be back soon" - It also takes AGES to show me my skills when I log in… :frowning:

The VM’s have no problem with internet access, and I am accessing the home.mycroft page from the host machine.

I am located in the UK BTW,
Great project!

Just to add to this…
it seems to be ONLY the skills that won’t save at home.mycroft.ai, as if I go to “advance settings” and change something, and tell it to save , I get the “all saved” message OK.

Weird??

Aaaaand… Now the error I am seeing in skills on home.mycroft have gone away LOL! - it had been doing that all day for me till now :slight_smile:

I am still seeing the timeout errors on the CLI though, is that normal?

Thanks for pointing this out. We are experiencing an issue with the skills service. All other services are working normally. You can get live updates as we work on this issue here.

As you’ve noticed, skill settings fail to load around 60% of the time. The timeouts you see in the CLI are from the failed requests to the backend. While changed settings may not load properly, everything else should still work. Thanks for your patience as we resolve this issue :).

Thanks :slight_smile:
I was just wondering if something weird was happening, or if it was normal … It must be the weekend LOL!

Have a good one!

Hello again.
I have come to a realisation , after reading other issues here that there is a problem at the mycroft.ai end, and it relates to this issue.
IMHO - your DDOS protection is stopping users who have devices set up!!!

I don’t know how often the devices poll ( often… ) but I cant hit the skills service from my PC and the skills etc. wont update on the devices running here… BUT they DID update after I had it all off for a while…
What is common?

  1. The same originating IP address,
  2. and stopping requests for a while… ( a few hours)

My guess ( and it is only a guess!) is that when there are too many attempts to connect the (upstream?) DDOS protection is black holing the IP’s until there is no traffic for some predetermined time…
it ALSO fits the “friday night was worse” info in the ticket! - It also would be worse as people set up new versions, and would stop new setups from getting their skills in the first place.

Please double check this - it may not be internal to your systems but a (paid) function of your ISP

While I am experiencing this same Read time out issue (on a Mark 1, v18.2.0), I am not so sure it is a DDOS protection issue. I tried disconnecting Mycroft from the network for over a day (but not shutting him down), within a minute of reconnecting, he attempted and failed to retrieve settings, same error message.

16:59:41.693 - mycroft.skills.settings:_poll_skill_settings:401 - ERROR - HTTPSConnectionPool(host='api.mycroft.ai', port=443): Read timed out. (read timeout=15)
16:59:41.695 - mycroft.skills.settings:_poll_skill_settings:402 - ERROR -
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.0-py2.7.egg/mycroft/skills/settings.py", line 398, in _poll_skill_settings
self.update_remote()
File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.0-py2.7.egg/mycroft/skills/settings.py", line 372, in update_remote
skills_settings = self._request_my_settings(hashed_meta)
File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.0-py2.7.egg/mycroft/skills/settings.py", line 440, in _request_my_settings
settings = self._request_settings()
File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.0-py2.7.egg/mycroft/skills/settings.py", line 456, in _request_settings
"path": self._api_path
File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.0-py2.7.egg/mycroft/api/__init__.py", line 53, in request
return self.send(params)
File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.0-py2.7.egg/mycroft/api/__init__.py", line 78, in send
data=data, json=json, timeout=(3.05, 15))
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 499, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='api.mycroft.ai', port=443): Read timed out. (read timeout=15)

@a_user, is that just exactly how your error log lines look in mycroft-cli-client ? or is it a little different from that?

Yep, that is the same!
I am not too sure about my theory now, but I have not been able to get to the “your skills” section on the hom.mycroft.ai pages at all for days now… not even when I took my PC to a different city :frowning:

Same here. I forgot to mention that as well. “Your Skills” section of home.mycroft.ai never loads “your skills”. Not sure if it is related, but the Location property on the devices tab does not seem to be filled out or persist when I attempt to set it.

I am having similar issues connecting to api.mycroft.com. When I try to login to home.mycroft.ai with my Github credentials, I get http code 504 (Gateway Timeout) and my new mycroft installation is unable to connect to api.mycroft.ai to complete activation. Has anyone found a workaround to get devices functional?

Same here, I am unable to login to home.mycroft.ai to activate a new device, has been broken for several hours now. If you folks @ Mycroft need an expert to consult with you about proper distributed DNS, network load balancing, and Linux clustering, so that these problems are resolved and never happen again, please let me know, and I’ll get you in touch with the right people. It’s very damaging to the community to have issues like this, which can be eliminated when things are configured properly on the back-end. Thanks.

1 Like

We’ve had a major outage over the last couple days, this has now been resolved.
You can read the open and transparent incident report here: