Issues with hkdata.fairuse.org?

[awdspyder]

I've been receiving errors when attempting to download SDF info since Friday, 2/18/2021. I assumed this was just a temporary server outage, but now I'm not so sure. I checked my Sophos UTM logs just to make sure there was no funny business going on with the web filter and the log indicated something interesting:

url="https://hkdata.fairuse.org/" referer="" error="Host not found"

Subsequent DNS lookups appear to be hitting an Amazon load balancer:

nslookup - 1.1.1.1
Default Server: one.one.one.one
Address: 1.1.1.1

> hkdata.fairuse.org
Server: one.one.one.one
Address: 1.1.1.1

Non-authoritative answer:
Name: prod-sav-park-lb01-1919960993.us-east-2.elb.amazonaws.com
Addresses: 2600389af43:7b57:5364:6ed6
26003894f8953cf:2439
260038949a7:5dd5:478b:9e4e
3.19.22.30
18.188.92.228
52.14.33.37

digs on Cloudflare, Google or OpenDNS all return a status of "non-existent domain." MXToolbox lookups indicate the same:

dig +time=5 +retries=0 +tcp +nocmd +nostats @1.1.1.1 a hkdata.fairuse.org
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54038
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hkdata.fairuse.org. IN A

;; AUTHORITY SECTION:
fairuse.org. 3600 IN SOA ns1.afraid.org. hostmaster.becnel.net. 2202200315 86400 7200 2419200 3600

Previous info indicates that hkdata.fairuse.org was located at 91.224.23.225, but obviously this could have changed in the past several months. Regardless, updating the local hosts file entry with this IP had no impact.

Reverse lookups on that IP point to a server that's clearly in Russia:

ping -a 91.224.23.225
Pinging mskf23-225-v.komtet.ru [91.224.23.225]
Reply from 91.224.23.225: bytes=32 time=121ms TTL=40
Reply from 91.224.23.225: bytes=32 time=121ms TTL=40
Reply from 91.224.23.225: bytes=32 time=122ms TTL=40
Reply from 91.224.23.225: bytes=32 time=121ms TTL=40

Interestingly, attempts to resolve mskf23-225-v.komtet.ru fail with a status of "non-existent domain" as well:

dig +time=5 +retries=0 +tcp +nocmd +nostats @1.1.1.1 a mskf23-225-v.komtet.ru
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 38451
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;mskf23-225-v.komtet.ru. IN A

;; AUTHORITY SECTION:
komtet.ru. 3600 IN SOA panel.komtet.ru. admin.komtet.ru. 2017071379 3600 3600 604800 86400

Anyone else in the U.S. experiencing this behavior?

[mike admin]

I've been receiving errors when attempting to download SDF info since Friday, 2/18/2021. I assumed this was just a temporary server outage, but now I'm not so sure. I checked my Sophos UTM logs just to make sure there was no funny business going on with the web filter and the log indicated something interesting:

MakeMKV should download SDF and all keys automatically, without the need for this URL. If it does not, please post/send the log and check porxy settings if you have any.

If you are looking just for a latest SDF.bin , you can always get it at http://www.makemkv.com/sdf.bin (this is just sdf.bin , no keys of any type are hosted at makemkv.com) - but again, you should already have it.

There is no need to download anything by hand from hkdata.fairuse.org and this url can go away any moment.

[Woodstock]

Mike, we'll need to update the fixing unknown key and UHD FAQ sticky messages then, since both refer to this URL when troubleshooting.

[Billycar11]

I'm not sure if this is a server issue or a people issue but have been getting emails about it I have updated the guide my sdf from 2/22/22 all you do is replace your privatedata.tar with mine and you will have an updated one if you are unable to get it at all.

I think it's as Mike says maybe makemkv has swapped to a new url and lot of people have manually white listed this one and now need to whitelist the new one because they have geoblocks and junk.

[awdspyder]

Hey Mike,

First, thanks for a great product; I've been a longtime supporter.

I am aware that MakeMKV will automatically download the SDF file and keys. I also wasn't trying to download anything by hand. I was simply mentioning that URL as a point of reference while troubleshooting what appeared to be a widespread DNS issue. Are you intimating that the application does not reference this by DNS name somewhere when attempting to download the keys? I have evidence to the contrary, and frankly, how else would it do it? I also think my troubleshooting steps may be misunderstood:

The application (MakeMKV) stated it could not retrieve the SDF file for several days. I examined my Sophos UTM logs to troubleshoot the problem.

I also didn't get the URL from the forum - I got it from my own local troubleshooting. The firewall logs referenced the destination by URL (not IP), indicating the application is trying to hit this URL by DNS name. It couldn't have even been associating the IP to an A record/CNAME via reverse DNS because the PTR record points to mskf23-225-v.komtet.ru. I could easily reproduce the issue as well - each time I'd start MakeMKV and it would error on the SDF/keys download, and the firewall would generate a corresponding log entry indicating the hkdata.fairuse.org domain was non-existent.

Further troubleshooting indicated a much more widespread domain name resolution issue, at least in my part of the US (I tried from multiple disparate networks/providers/machines that were 100+ miles apart).

Additionally, I have nothing locally whitelisted with respect to MakeMKV, nor am I performing GeoIP blocks of any kind, so a change in IP is irrelevant if the DNS records are properly updated and working. Apparently, however, there's something going on in Russia right now? Wouldn't surprise me in the least if this was related, but perhaps it was just an innocuous Internet idiosyncrasy (my college e-comp instructor would have been proud of that alliteration).

Regardless, it is now completely resolved in MakeMKV. Traditional DNS lookups from my local DC as well as digs against Cloudflare/Google appear to be working again as well.

Thanks!

[awdspyder]

I think it's as Mike says maybe makemkv has swapped to a new url and lot of people have manually white listed this one and now need to whitelist the new one because they have geoblocks and junk.

I do not think this is the case. If MakeMKV had programmatically changed to a new URL this would be evident in the firewall logs. The logs indicate that each time MakeMKV starts (or perhaps at disc load, haven't looked at the exact timing), it hits hkdata.fairuse.org (not saying that's the ONLY thing it hits).

Besides, how could it have changed? I hadn't updated MakeMKV recently at that time, which is what would be required for MakeMKV to use a new URL, unless the URL is embedded in a file that it's downloading from yet another URL (the sdf.bin perhaps?). Frankly, it's also much more common to perform a DNS change (i.e. new IP) than to update to a completely new URL - it's kinda the whole point of DNS.

At any rate, it's working fine now, and still at the original IP of 91.224.23.225, so nothing appears to have been changed or updated.

[Woodstock]

The location to access is included in the sdf.bin file, which has been changing a LOT recently, as support for the Pioneer drives has been added.

But, as you say, the hkdata URL resolves again.

[inlandchris]

Having issues trying to find the IP address that has to do with hkdata.fairuse.org.
I have narrowed it to Russia with these IP address that I have in my windows Defender to block all Russian sites:
185.81.220.0/22
185.81.248.0/22
185.82.12.0/22
185.82.24.0/22
185.82.176.0/22
185.82.196.0/22
185.82.244.0/22
185.83.240.0/22
185.83.242.0/22
185.84.32.0/22
185.84.40.0/22
185.84.108.0/22
185.84.162.0/22
185.84.172.0/22
185.84.162.0/22
185.84.184.0/22
185.84.244.0/22
185.85.122.0/22
185.85.160.0/22
185.85.216.0/22
185.86.92.0/22
185.86.112.0/22
185.86.216.0/22
185.86.132.0/22
185.86.144.0/22
185.86.48.0/22
.... to
185.105.224.0/22

too much to list
if any of these match, please reply
thanks

[segfaulted]

The exact IP is 185.84.108.20 as provided in this thread
viewtopic.php?p=141837#p141837

[inlandchris]

Many thanks!