FW 3.2.7 breaks custom web certificates

[alxwolf]

          my internal CA is as follows:

Root CA
Intermediate CA
Router certificate

the router certificate is signed by my intermediate CA.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            83:a3:ae:9a:21:e4:00:4c:d2:89:5a:99:f9:41:fb:ad
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O = PaulGG Intranet, CN = PaulGG Intranet Intermediate CA
        Validity
            Not Before: Dec 14 12:04:01 2023 GMT
            Not After : Dec 15 12:05:01 2023 GMT
        Subject: CN = router.paulgg.int
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:a2:dc:42:d7:35:b3:dd:5b:96:86:29:4c:64:e7:
                    32:48:1c:0a:bb:e6:5f:43:ad:63:de:19:12:3e:3f:
                    a5:52:f3:20:1e:30:23:c4:ac:56:aa:d8:b8:5d:ae:
                    d8:fe:6b:e8:ce:c5:b6:e6:c8:20:36:e0:98:35:3f:
                    b5:c3:d3:51:cc:91:ca:1d:38:a7:c8:9c:9e:7b:9a:
                    5e:41:e1:5b:38:7e:b8:7d:6e:a3:d7:51:11:b9:c7:
                    1c:be:f7:44:b2:a9:f5:63:1a:f3:43:64:ea:11:f1:
                    97:4b:6b:a6:e4:95:e1:05:98:45:4f:63:8b:ae:ac:
                    8d:aa:7d:9b:a5:d8:b8:1d:61:5e:c1:9b:3a:d8:91:
                    45:57:a0:d6:b7:1c:d4:88:4e:01:e5:c3:61:b4:6d:
                    68:1d:c8:2c:00:9a:e8:fb:0c:76:f1:f0:b8:d2:22:
                    3b:35:c8:9c:b9:80:db:7b:d9:ee:b3:af:b0:0d:fc:
                    70:b8:ef:0b:f6:4c:39:b5:c8:d0:9c:86:b3:71:a1:
                    a5:42:26:f7:3e:82:4c:23:a0:84:67:f6:0a:a9:34:
                    eb:ac:f2:f6:bc:a5:f5:6e:1a:34:e8:2a:a6:7e:42:
                    90:28:b9:db:34:1d:0d:b7:a1:1a:65:d3:61:55:ad:
                    ec:52:0a:fe:f5:11:41:1d:21:12:25:fc:02:e8:4d:
                    9d:cd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:58:1E:16:42:18:5A:CF:A5:DA:29:2E:37:50:AF:4E:5D:1B:CE:B9:93

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                98:41:E0:9B:1A:F8:23:F4:8F:6C:D7:53:37:A4:4E:D6:79:D3:A9:57
            X509v3 Subject Alternative Name:
                DNS:router.paulgg.int
            1.3.6.1.4.1.37476.9000.64.1:
                [email protected].+V5B126r4pfQJE_PkrMvSvc_gFds2wEs5EhzVSdSF-dQ
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:be:81:ea:60:62:19:71:3e:0b:54:48:43:79:
         56:5f:b5:59:23:dc:21:1b:94:16:07:a6:74:f6:5a:24:ff:af:
         43:02:20:25:29:02:8e:64:0f:01:b0:4e:b8:3d:47:8c:c5:b1:
         f1:80:46:86:fd:48:55:df:65:fb:af:fb:2e:24:eb:8f:a9

I just checked /etc/ssl/certs and looks like there's some stuff that has been deleted, including my root certificate from the keystore.

When I have some more time I'll play around with this...

Originally posted by @therealpaulgg in #61 (comment)

[alxwolf]

@therealpaulgg opened a new issue, your case is different from the RADIUS issue many people seem to see.

You debugged that your root certificate is gone from keystore - that may be another hint in the direction that UniFiOS says "incorrect / incomplete chain -> trash it, put self-signed in".

but as you wrote, root cause for trust being broken in cert store is unclear...

I assume when you copy the certs manually and restart unifi-core, they get culled, too?

[therealpaulgg]

So I mentioned in my last comment that I ended up trying to fix my cert store manually by both re-adding my Root CA and intermediate CA (I didnt need this previously)

ran openssl verify and it reported OK

Yeah copied the certs manually and they are gone.

Maybe if I try configuring RADIUS (I never bothered) it'll work?

[therealpaulgg]

This one is a bit trickier to solve than the RADIUS one. Still not entirely sure what to do.

I did some digging based on what @ouaibe had mentioned in the other issue, and I was only able to find this one entry:

"unifiNetwork": {
   "certificate": {
    "crt": "CERT_HERE"
   },
   "controllerURL": "https://192.168.1.1:8443",
   "enabled": true,
   "informURL": "http://192.168.1.1:8080/inform",
   "sitename": "default",
   "uciAllowList": []
  },

this CERT_HERE is also a base64 encoded value but does not decode the same way that the radius certificates do, so I am a bit confused.

Tried searching for other configuration files and had no luck. There is one called /data/udapi-config/udapi-net-cfg.json but that appears to just be a subset of the previously mentioned /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state.

Its like the implementation of RADIUS and the web certs are just completely different, and I'm not able to find where they are configured at.

[ouaibe]

Looking at it, that certificate is in DER format (binary), not PEM. These are both interchangeable/convertible using the following commands:

DER -> PEM
openssl x509 -inform der -in certificatename.der -out certificatename.pem

PEM->DER
openssl x509 -outform der -in certificatename.pem -out certificatename.der

So in theory, if you already have your local CRT file server.crt in PEM format, you can do something like this:

jq --arg content "$(openssl x509 -outform der -in server.crt | base64)" '.services.unifiNetwork.certificate.crt = $content' /data/udapi-config/ubios-udapi-server/ubios-udapi-server.state > /data/udapi-config/ubios-udapi-server/ubios-udapi-server.new.state

Then validate the NEW file, the JSON content at .services.unifiNetwork.certificate.crt and if you're satisified with it (having taken a backup) you can replace the current .state file with the new one, and finally restart the udapi-server.

⚠️ I haven't tested this, tread carefully!

[therealpaulgg]

This makes sense... only concern I have is where is the key? previously for web certs it needed both the cert and the key. You should need both pieces, right?

[ouaibe]

A CRT file for a server can (should?) contain both the cert and the private key. You could try generating a regular CRT file in PEM format, that would contain the two sections:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
... 
-----END PRIVATE KEY-----

And then convert that to DER and see if it works?

Alternatively, you could try hacking a key="value" element in the JSON file similar to how the other certs are setup, and see if udapi-server breaks while booting?

[ouaibe]

After digging a bit more, the unifiNetwork certificate in the JSON config file has the exact same fingerprint than the one in the keystore at /data/unifi/data/keystore.

Compared the output of openssl x509 -inform der -in unifinetwork.der -noout -fingerprint and the output of keytool -v -list -keystore /data/unifi/data/keystore -storepass aircontrolenterprise.

This might mean that the keystore is used to back the signature for that part of the certificate management system.

FWIW you can export the existing private key in PEM format using the following commands:

keytool -importkeystore -srckeystore /data/unifi/data/keystore -srcstorepass aircontrolenterprise -destkeystore /tmp/keystore.p12 -deststoretype PKCS12 -deststorepass aircontrolenterprise

openssl pkcs12 -in /tmp/keystore.p12 -nodes -nocerts -out /tmp/private_key.pem

But I'm not sure how that private key would be useful for you, instead you could try adding your own private key/certificate to the existing keystore and see how it behaves with udapi-server.

Good luck!

[therealpaulgg]

Unfortunately it seems this certificate is completely different from the one on the web portal. Fingerprints dont match at all and the issuers are totally different (CN = unifi.local for web cert vs. C = US, ST = New York, L = New York, O = Ubiquiti Inc., OU = UniFi, CN = UniFi)

I found a log in /data/unifi-core/logs/http.log saying:

2023-12-18T21:15:19.493Z - info: Self signed certificate needs regenerated, regenerating
2023-12-18T21:15:19.494Z - info: Generating /data/unifi-core/config/unifi-core.crt

wonder if the underlying cert implementations for unifi-core and the other stuff are just totally different.

[ouaibe]

Ah, reading your message I had assumed the CRT in the JSON file was the right one, sorry to have mis-interpreted that.

It's very likely they combine 3+ certificate management mechanisms, it seems to be all over the place between freeradius, the keystore (that must be used for something, maybe intra-Unifi comms over HTTPS), some certs for Protect and the unifi-core/Web UI stuff...

One thing you "could" try, maybe to get more output out of udapi-server boot sequence is to chattr +i the unifi-core.crt file. It is very likely they're not checking for that so either it makes it immutable across reboots as a temporary solution for you, or crashes udapi-server and/or you get more info in its journalctl -u because it cannot replace the file...

Either way, if they are re-generating it, the private key and the cert must be somewhere...

[pokrface]

One thing you "could" try, maybe to get more output out of udapi-server boot sequence is to chattr +i the unifi-core.crt file. It is very likely they're not checking for that so either it makes it immutable across reboots as a temporary solution for you, or crashes udapi-server and/or you get more info in its journalctl -u because it cannot replace the file...

After being frustrated by 3.2.9's continuation of 3.2.7's bad behavior, I found this thread. I can confirm that marking both the cert and key immutable prevent them from being replaced upon restarting the unifi-core service. It's a kludge, but it'll do until a real fix comes along!

[therealpaulgg]

One thing you "could" try, maybe to get more output out of udapi-server boot sequence is to chattr +i the unifi-core.crt file. It is very likely they're not checking for that so either it makes it immutable across reboots as a temporary solution for you, or crashes udapi-server and/or you get more info in its journalctl -u because it cannot replace the file...

After being frustrated by 3.2.9's continuation of 3.2.7's bad behavior, I found this thread. I can confirm that marking both the cert and key immutable prevent them from being replaced upon restarting the unifi-core service. It's a kludge, but it'll do until a real fix comes along!

Absolutely genius workaround...this totally solves my problem. Finally got around to actually implementing it on my Unifi router and its pretty straightforward. It is a little more complicated since I'm using step-ca which renews certs daily, but it was easy enough to modify the renewal service to simply make the file temporarily mutable and then make it immutable once renewed.

Thank you very much, I had pretty much given up on the idea of accessing Unifi from a trusted domain at this point.

[forrestrae]

Not sure this helps, but I'm on EA release v3.2.12 and all I had to do was place move the cert and key to /data/eus_certificates/unifi-os.crt and /data/eus_certificates/unifi-os.key respectively.

[alxwolf]

@forrestrae interesting observation, have you ever used Glenn R.'s "UniFi Let's Encrypt" before?

That script creates such directory. if you did not use it before, maybe UI has implemented that scripts mechanisms into firmware.