Hmm, that's not my understanding of how a UNIX-based operating system behaves... Can you confirm that MakeMKV is actually using the KEYDB.cfg file please, i.e. by moving the file out of the data directory, restarting MakeMKV and trying to decrypt your disk again? It occurs to me that MakeMKV might also have added your key to its hashed keys.
I know it's been a couple weeks, but just to clarify. By default disks are not formatted as case sensitive on Mac OS. It's possible to do so (at least with APFS and IIRC with HFS+ as well) but it's not recommended unless you have a very specific need for it. The default formatting will respect whatever case you want to use but "KEYDB.cfg" and "keydb.cfg" would not be considered unique file names and you could not have both of them in the same directory.