Support road.cc

Like this site? Help us to make it better.

Garmin Connect ransomware attack – site still down after 24 hours

US firm yet to officially confirm cause of downtime with several of its other services also hit

Yesterday we reported how Garmin users throughout the world have been unable to access the US firm’s Connect service through which rides and runs are uploaded from devices. While the company has yet to confirm that it has been hit by a ransomware attack, several employees are reported to have said this is the case on social media.

Garmin Connect went offline early yesterday (Thursday).

Those signing into the Garmin Connect site are currently greeted by a message reading: “We are currently experiencing an outage that affects Garmin.com and Garmin Connect. This outage also affects our call centers, and we are currently unable to receive any calls, emails or online chats. We are working to resolve this issue as quickly as possible and apologize for this inconvenience.”

ZDNet reports that a ransomware attack has encrypted its internal network and some production systems.

As well as its official website and Connect data-syncing service, the issue is also said to be affecting Garmin's aviation database services and several production lines in Asia.

This includes flyGarmin, a website that supports the company's aviation navigational equipment. Some pilots have therefore been grounded as they are unable to download an up-to-date version of the firm’s aviation database which is a requirement to fly.

Garmin has thus far refused to comment on suggestions that the issues have been caused by ransomware, saying only that it is investigating. This means that it is currently unclear whether any customer data has been lost or stolen.

While many features on Garmin devices cannot currently be used, data saved on them has not been lost.

Rides recorded on a Garmin smartwatch or bike computer will remain on your device unless you delete them.

They can also be posted to services such as Strava manually by connecting to a computer via USB, downloading the .fit file from the activities folder, and then uploading it from the computer to the website.

On Strava, there's an option to upload activities manually from a drop-down menu that appears on the top right on the desktop site, and on the top left on their mobile app.

Alex has written for more cricket publications than the rest of the road.cc team combined. Despite the apparent evidence of this picture, he doesn't especially like cake.

Add new comment

36 comments

Avatar
hawkinspeter | 4 years ago
0 likes

According to ZDNet, they're suffering the effects of WastedLocker: https://www.zdnet.com/article/garmin-services-and-production-go-down-after-ransomware-attack/

Avatar
brakesmadly | 4 years ago
2 likes
Avatar
brooksby | 4 years ago
2 likes

This one time (at band camp...) we got hit by a ransomware thing.

I'd wondered why our server was so active and the network seemed so slow.

Physically walked over and logged into the main server (yes, it was an interface just like in Alien!), checked processes, checked the file structure, and could actually see files being encrypted!  Horrible experience.

Management took the view to just nuke the whole thing and start over - had all sorts of data backups, but it was a real eye opener.

Avatar
Llewelyn | 4 years ago
1 like

Please! Can anyone tell me why a world-class, worldwide company doesn't have a complete backup system?
My mind boggles at the lack of insight by the computer men of the company.

Avatar
FlyingPenguin replied to Llewelyn | 4 years ago
11 likes

They probably do, and if it was something non-malicious (e.g. a power issue at a data center) then no problem, switch to the second site and continue, maybe a few minutes down time, hours tops.

In a malicious scenario this gets more complicated for two broad categories of reasons:

1) You've got to actually be confident you have located the source of the issue, second sites are usually connected to the primary (for data replication) and other offices (for maintence/control), as well as the internet (to serve customers).  If your second site isn't impacted already, simply failing over before being sure you've fixed the issue could be enough to allow the infection to spread.

2) Depending on the attack, the main payload (the encryption logic in the case of ransomware) could have been dormant for some time already whilst it spread, meaning that your backups may already contain corrupt data and/or dormant copies of the virus.  Restoring from backup may reintroduce the malware into the network environment.

Either way, in the case of a malicious scenario, the sensible thing to do is usually to fix the root cause of the problem before trying to stand the systems back up, even if they do have a "complete backup system".  They aren't critical national infrastructure, there is no overriding reason to wing it.

Avatar
kil0ran replied to FlyingPenguin | 4 years ago
0 likes

The secondary site could be critical for them. For Maersk they ended up sending an engineer to a site office in Ghana where a single solitary Domain Controller was still alive and unencrypted (a domain controller is the bible for the entire IT infrastructure - every user, every device, security and so on). Without that fortunate accident they would have been weeks longer in restoring service

Avatar
hawkinspeter replied to Llewelyn | 4 years ago
7 likes

Llewelyn wrote:

Please! Can anyone tell me why a world-class, worldwide company doesn't have a complete backup system?
My mind boggles at the lack of insight by the computer men of the company.

It won't be the computer people, it'll be the accountants.

It's cheaper to not have a decent backup system until the time when it suddenly becomes a lot more expensive.

Avatar
kil0ran replied to hawkinspeter | 4 years ago
1 like
Avatar
hawkinspeter replied to kil0ran | 4 years ago
1 like

kil0ran wrote:

Evidence for the prosecution m'lud

https://www.networkworld.com/article/3200105/british-airways-outage-like...

Some basic rules that should be followed:

  • You don't have a disaster recovery plan until it has been tested
  • You don't have backups until you've tested them
  • If you don't know what happens when equipment is powered off and on again, then it's not production-ready
  • There's 2 types of people - those who have lost data and those that will lose data
Avatar
Hirsute replied to hawkinspeter | 4 years ago
0 likes

One of our software vendors told us of one client who had a really good backup and recovery which they had tested. Then they had a flood in the basement and the only servers they had were in the basement...

Avatar
hawkinspeter replied to Hirsute | 4 years ago
1 like

So, they just bought new servers and restored from the off-site backup - job done!

Or do you mean that they only had on-site backups? (In the one location that is sensitive to floods)

Avatar
Hirsute replied to hawkinspeter | 4 years ago
0 likes

I think the backup was offsite, but obviously they had nowhere to restore to.

I guess that they did buy new servers - but an expensive option and they would need a bit of configuration and additional vendor support for each system.

The moral is you don't know how good your back up is until crunchtime, no matter how good you think it is .

Avatar
kil0ran replied to Llewelyn | 4 years ago
0 likes

They do. It's also encrypted by the ransomware, or potentially they don't have any kit for the engineers to get access to it.

They've got two options:

1. Pay the ransom

2. Rebuild from scratch

It took Maersk (global logistics/shipping) two weeks to restore service when they were ransomed, Garmin don't have the same resources. Good long read here

https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-cra...

Avatar
risoto | 4 years ago
0 likes

I saw that the attack could perhaps end up encrypting our data on Garmin devices. If you want your data back its about 2,500 pounds....!

Time to dump Garmin and start to enjoying the ride perhaps.

Avatar
Zebulebu | 4 years ago
0 likes

Absolutely atrocious lack of communication - which is standard for Garmin. Even if they don't know what's been stolen/lost, at least tell people the cause and advise them what data COULD have been stolen as a result so that they can make arrangements to try and protect themselves (eg: credit card information if they store it unencrypted somewhere, passwords for the same reason, location data etc)

Pathetic

Avatar
mdavidford replied to Zebulebu | 4 years ago
5 likes

If it is a ransomware attack, then typically the data isn't actually stolen - it's just encrypted in place.

Avatar
brooksby replied to mdavidford | 4 years ago
0 likes

Which is as good as stolen (to Garmin - 'lost', anyway), unless they pay the ransomers a lot of money...

Avatar
mdavidford replied to brooksby | 4 years ago
3 likes

Yes - but it's not the same as 'stolen' as in users' credit card details and personal information being in the hands of the attackers, which is what Zebulebu seemed to be worried about.

Avatar
brooksby replied to mdavidford | 4 years ago
0 likes

True; fair enough.

Avatar
FlyingPenguin replied to mdavidford | 4 years ago
3 likes

Historically, yes, however over the last 12-18 months there's been an increase in hybrid attacks where data is both exfiltrated and then encrypted (e.g. Maze ransomware).

It's entirely possible they don't yet know what has been exfiltrated (if anything) so simply saying "they could have got to anything we hold, but we don't know yet" helps no one. 

As a couple of people have said, this is not just "Oh a system fell down, fail over to the DR site", you need to make sure you eliminate all instances of the malware and also make sure you aren't simply restoring corrupted backups that include the virus, otherwise you're straight back to square one.  That is substantially more complicated than your run of the mill fail over scenario.

Avatar
risoto replied to mdavidford | 4 years ago
0 likes

What is the difference? They steal your credit card and ask you to pay a huge amount to get it back?

Avatar
mdavidford replied to risoto | 4 years ago
2 likes

risoto wrote:

What is the difference? They steal your credit card and ask you to pay a huge amount to get it back?

The difference is they haven't stolen it*. It hasn't gone anywhere - it's still sitting on the victim company's servers, but just encrypted so it's unusable to them. You don't need to pay anything to get it back - you already know what your credit card details are. It's the company they demand payment from.

[* Unless, as FlyingPenguin said, it's not just launched a ransomware attack, but has also extracted the data. One would hope, though, that however flawed Garmin's systems were, they would at least have encrypted credit card / personal data, and that encryption should mean it would still be useless to the attackers**. Even if they hadn't, paying to get get your card details back would be pointless - you already have them, and there's nothing to stop the attackers keeping a copy, so you'd just cancel the cards.]

[** At least until quantum computing comes along, and then we're all screwed.]

Avatar
Awavey replied to mdavidford | 4 years ago
0 likes

Maybe so but if the ransomers have managed to gain access to Garmins internal IT setup so successfully to encrypt it on such a scale it basically borks all of their internal & external facing IT, and these things are usually firewalled and kept distinctly separate for a good reason,theres no telling what theyve peaked at,copied,or left behind in the time they had unrestricted access. This isnt someone accidentally loaded a virus from a floppy drive or clicked a bad link in an email.Disaster recovery shouldn't take more than 24hrs on a production environment,even a global one.

Avatar
Pedal those squares replied to Awavey | 4 years ago
6 likes

You obviously have had little dealings with such attacks.   Once they are in, the virus jumps from machine to machine encypting as it goes or in some cases only encypting a percentage of devices, leaving some users thinking they are ok, but in the background their machine keeps connecting to other machines and infecting them. 

It is not a question of just getting a tape out of the cupbaord and pressing restore.

You may have to go to each machine in turn, disconnected it from the network and then run fixes on each machine.  In a global compaine that can take days and days.  Only then can you trun on the whole network again.

No matter who you are, once you connect any part of the network to the outside world via the internet, USB, Disc, etc....your network can never be 100% secure, it is just impossible.

Judge them by how they deal with the situation they find themselves in.  If they did have rubbish security, their systems would be up and down all the time with issues, but in my experience that has not been the case.

 

Avatar
Awavey replied to Pedal those squares | 4 years ago
1 like

and youve obviously watched far too many films and tv if you think thats how easy it is to spread malware across a whole office IT environment including voip call servers and a production front end level IT setup which would be specifically hardened for genuninely state sponsored hacking attacks, not spotty teens in their bedroom mining bitcoins

but who knows, I dont claim to know their network setup, I know Id never advise anyone who sought my opinion to allow that to be remotely possible precisely because its such an obvious flaw in their security and the damage it could cause, but maybe they are that naive to link them up so you could wipe it all out if they listen to obvious security expert con-sultants like yourself

Avatar
rswift replied to Awavey | 4 years ago
0 likes

Awavey wrote:

and youve obviously watched far too many films and tv if you think thats how easy it is to spread malware across a whole office IT environment including voip call servers and a production front end level IT setup which would be specifically hardened for genuninely state sponsored hacking attacks, not spotty teens in their bedroom mining bitcoins

but who knows, I dont claim to know their network setup, I know Id never advise anyone who sought my opinion to allow that to be remotely possible precisely because its such an obvious flaw in their security and the damage it could cause, but maybe they are that naive to link them up so you could wipe it all out if they listen to obvious security expert con-sultants like yourself

Get an over sized fizzy drink and some popcorn... have a read: https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-cra...

Avatar
mdavidford replied to Awavey | 4 years ago
5 likes

Awavey wrote:

if the ransomers have managed to gain access to Garmins internal IT setup

They may not have needed to - they may well have used someone who already had privileged access to the systems as a vector to introduce the malware, and it then spread itself from there.

There may also be a precautionary 'lockdown' aspect to everything being down as well - if you haven't yet identified how the infection spread, then it may be better to take everything down until you do, rather than bring affected systems back and risk re-infection from elsewhere (a 'second spike', if you will).

Avatar
Awavey replied to mdavidford | 4 years ago
1 like

which is possible sure, but companies who take this stuff seriously, and Id have assumed Garmin would fall under that label but maybe they dont, would never have a setup that allowed malware that got into their office IT environment, which is identified in all security audits as the number 1 likely vector threat risk, to then propogate across into the production environment.

Because everyone knows thats the biggest risk of bringing down a live server, excepting asteroid strike,fire or flooding,

So you dont go nah it will be ok, you lock down production networks to the nth degree from office IT, never allow access from an untrusted domain, certainly never allow code deployments from them and then so what if your internal IT team is Fubar'd by malware, you wont have impacted any live customer data or access, you might not be able to deploy updates till you fix it, but youll still have a service.

Even if your live server did go down as a result, your disaster recovery process should have an alternate server,in an alternate secure location with a safe/secure back up spun up within 24hrs maximum downtime, and thats being generous most IT service level agreements insist on 3hrs max for live downtime recovery, beyond that it starts getting increasingly expensive in compensation.

Avatar
FlyingPenguin replied to Awavey | 4 years ago
4 likes

Awavey wrote:

which is possible sure, but companies who take this stuff seriously, and Id have assumed Garmin would fall under that label but maybe they dont, would never have a setup that allowed malware that got into their office IT environment, which is identified in all security audits as the number 1 likely vector threat risk, to then propogate across into the production environment.

No one can 100% guarentee that.  Major government agencies and financial institutions with ooodles of resources (and regulators breathing down their necks) are still regularly breached, it's a question of when, not if, for a company like Garmin.

You can introduce controls with reduce the likelihood of compromise (e.g. firewalls, internal segmentation, IDS/IPS, SIEM/network activity monitoring & endpoint protection like AV and DLP), or reduce the impact (e.g. comprehensive backups, playbooks for major scenarios, incident response support retainers), but once the malware is past the external permieter you are reliant on the controls you do have in place preventing travesal of the internal network into production systems.  Unless your production system is air gapped from everything (including the external internet, which wouldn't be appropriate here), you have to be right for every possible threat event, the attacker only has to be right once.  And honestly, having led numerous security audits, I've never not raised a finding...

It's entirely possible they did something stupid (WannaCry brought down large parts of the NHS using a vulnerability that was patched by Microsoft a long time before the attack, heads should have rolled), but to view the recovery time in terms of normal service SLAs of three or four 9s availability ignores the complexities of dealing with (potentially competent, potentially specifically targeted) malware.

Avatar
quiff replied to Zebulebu | 4 years ago
4 likes

I imagine it's very difficult to communicate meaningfully to users while they are still investigating, working out what reports they need to make to regulators and law enforcement etc. 

Pages

Latest Comments