wowana.me
website source
commit 5b36b5cf81114c39b94da6dfb8e6d378bddf9692 parent 95f2999faa5dddba95853b2710322e213728340e Author: opal hart <opal@wowana.me> Date: Mon, 30 Mar 2020 08:03:34 +0000 blog post: "my experience with MivoCloud" Diffstat:
| A | src/blog/my-experience-with-mivocloud.md | | | 215 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 215 insertions(+), 0 deletions(-)
diff --git a/src/blog/my-experience-with-mivocloud.md b/src/blog/my-experience-with-mivocloud.md @@ -0,0 +1,215 @@ +# my experience with MivoCloud +<!--[time 202003300802.17]--> + +I just filed an issue today with [MivoCloud] [] after IPv4 connectivity +issues on my dedi. after debugging extensively, racking my brain, +wasting plenty of time, and consulting with my friend about it, I +realised that a subset of my /29 was pulled from under my nose and +reallocated to what I suppose is another customer. the ticket: + +> ## IPv4 addresses reassigned? (and other grievances) +> +> Hi, +> +> This time I'm sure the issue is outside of my control. Five of the eight +> IPv4 addresses I pay for in my 185.163.46.96/29 block are no longer +> routed to my server. I verified with nmap/tcpdump as below: +> +> ~ wowaname@mahin> doas nmap -sn -PE 185.163.46.138 185.163.46.96/29 +> doas (wowaname@mahin) password: +> Starting Nmap 7.70 ( https://nmap.org ) at 2020-03-30 06:05 -00 +> Nmap scan report for volatile.bz (185.163.46.138) +> Host is up (0.38s latency). +> Nmap scan report for 185-163-46-96.volatile.bz (185.163.46.96) +> Host is up (0.38s latency). +> Nmap scan report for mx1.volatile.bz (185.163.46.97) +> Host is up (0.37s latency). +> Nmap scan report for 185-163-46-101.volatile.bz (185.163.46.101) +> Host is up (0.38s latency). +> Nmap scan report for 185-163-46-102.volatile.bz (185.163.46.102) +> Host is up (0.38s latency). +> Nmap scan report for 185-163-46-103.volatile.bz (185.163.46.103) +> Host is up (0.37s latency). +> Nmap done: 9 IP addresses (6 hosts up) scanned in 1.74 seconds +> +> and on server +> +> # tcpdump -n -i bond0 '(icmp[icmptype]=icmp-echo or icmp[icmptype]=icmp-echoreply) and (dst 185.163.46.138 or dst net 185.163.46.96/29)' +> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode +> listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes +> 06:05:31.315509 IP [home IP] > 185.163.46.102: ICMP echo request, id 20075, seq 55153, length 8 +> 06:05:31.315726 IP [home IP] > 185.163.46.138: ICMP echo request, id 59515, seq 55157, length 8 +> 06:05:31.315772 IP [home IP] > 185.163.46.103: ICMP echo request, id 6724, seq 55155, length 8 +> 06:05:31.319262 IP [home IP] > 185.163.46.101: ICMP echo request, id 4617, seq 55150, length 8 +> ^C +> 4 packets captured +> 7 packets received by filter +> 0 packets dropped by kernel +> 1 packet dropped by interface +> +> The address <http://185.163.46.96/> points to something I definitely +> don't host, and traceroutes differ between 185.163.46.101 (which still +> routes to me) and 185.163.46.96 (which routes to someone else now): +> +> 16 ae0.cr3.ias.ro.mivocloud.com (185.163.44.121) 167.040 ms 196.535 ms 204.169 ms +> 17 ae1.cr1.kiv.md.mivocloud.com (185.163.44.122) 238.265 ms 252.588 ms 256.559 ms +> 18 volatile.bz (185.163.46.138) 256.777 ms 258.746 ms 269.423 ms +> 19 185-163-46-101.volatile.bz (185.163.46.101) 263.569 ms 267.788 ms 271.269 ms +> +> vs +> +> 16 ae0.cr3.ias.ro.mivocloud.com (185.163.44.121) 211.082 ms 211.650 ms 197.482 ms +> 17 ae1.cr1.kiv.md.mivocloud.com (185.163.44.122) 199.182 ms 198.986 ms 198.965 ms +> 18 185-163-46-96.volatile.bz (185.163.46.96) 198.910 ms 197.386 ms 198.901 ms +> +> So, I'm not going to lie, I'm kind of pissed. Mistakes happen, sure, but +> this was entirely preventable by tracking which IPs are already +> allocated to users. I pay for the /29 range because I do have a need for +> IPv4 addresses. I wish we could go IPv6-only sooner rather than later, +> but that isn't reality, so I do still need IPv4, and I can't get by with +> NATting all connections originating from my box. +> +> I did verify whether there was a potential for BGP hijacking but ARIN +> says Mivo's routing advertisements are signed using RPKI so I can safely +> rule that out. I've also verified that the issues I experience are not +> related to my home ISP: my friend has verified these issues I speak of, +> and random Tor exits seem to reflect my findings (e.g. accessing +> <http://185.163.46.96/> with Tor and without). +> +> Some questions I'll pose, and I would greatly appreciate full answers, +> as sysadmin to sysadmin, rather than as ISP to customer. I know I'm a +> customer but I go out of my way to pay for IaaS rather than anything +> managed, so I would rather know the infrastructure I'm working with. +> +> 1. Ranges for "Assigned IPs" on castellum clearly indicate +> `185.163.46.96/29` and `2001:67c:2db8:301::/64` on the [control +> panel] []. So, Mivo does have capability to track allocated IPs. +> What happened here so that those IPs seemed free to reuse for +> another customer? +> 2. Mivo's networking configuration has bothered me since day one. I've +> been willing to work with what I was given because otherwise Mivo +> provides a great deal with respect for alternate payment methods +> (cryptocurrency is the cash of the Internet moving forward, I +> believe, so I use it whenever I am able). But at some point I have to +> say enough's enough and ask Mivo to do their part in having +> acceptable networking to work with. So, could we work together to +> figure some of this out and save pain for the future? See the next +> question or so for details. +> 3. IPv4, I understand it's limited. That's why I chose pointopoint +> routing for the /29. I have a few pointed to VMs, containers, and +> physical machines over VPN. The routing is contrived for them though; +> I'd rather my gateway for these be directly to 185.163.46.129 (your +> gw) rather than to 172.22.3.1 and configure bird as so: `route +> 185.163.46.97/32 via 172.22.3.101; route 185.163.46.99/32 via +> 172.22.3.50; route 185.163.46.101/32 via 172.22.3.60;` – it wasn't +> too out of my way, I already had bird to participate in a shared VPN, +> but I still would rather my networking be more "correct". And I +> believe it's a good tradeoff; I have solely a /29 and I surrender a +> /32 back into Mivo's pool, makes it easier for all of us I believe. +> We can open a new ticket to track this and make sure we're prepared +> before switching IP configs over, for minimum downtime/disruption. +> 4. IPv6, there's *no* reason to be stingy about /64s. In fact I'm +> tempted to ask for a larger range just so I can use SLAAC properly +> through my network. I went with it when I first found out that Mivo +> doesn't give full /64s as promised on registration and purchase, and +> I'm glad you offered me a full /64 after asking, but I'd rather not +> route anything through 2001:67c:2db8:9::138/128 because it makes no +> sense I have one IPv6 from a /64. Just please give me +> 2001:67c:2db8:301::/64 routed directly to your gw so I can do away +> with an absolutely incorrect IPv6 setup. And like I said I'm tempted +> to ask for a larger range (/60 is most I'd need, to slice into +> sixteen /64s, good enough for current and future internal use) but +> I'll leave that thought on the table. I do a lot of virtual +> networking and plan to expand (see [Volatile] [], basically I want +> to offer services such as WireGuard VPN [with transparent Tor onion +> proxying which needs its own IPv6 range for hostmapping], mail, &c +> to donors and friends, to offset other cloud services like Google), +> and until I have enough out-of-pocket funds to build a server and +> colocate in the USA, I have to work with what I'm currently using. +> Also, I like suggesting Mivo to friends and anyone else looking for +> a host, and I hate telling them that "there's a catch with IPv6" – +> people expect an advertised /64 to work out of the box. +> 5. Unrelated to this ticket and I'll create a new one if necessary: but +> seriously, the current IPMI setup is bad for a few reasons: +> - Most obviously is the fact it's exposed to the Internet. IPMI +> firmware is a hassle to update from what I understand (IIRC +> requires upgrading the board itself?) thus outdated firmware is +> host to vulnerabilities: +> 1. [iDRAC 6 on CVE Details] [iDRAC 6] +> 2. [CVE-2018-15774 and CVE-2018-15776] [Dell CVEs] – notice how +> Dell explicitly recommends against exposing IPMI to the +> Internet +> 3. [iDRAC 7], [iDRAC 8], [iDRAC 9] all have at least one serious +> vuln, so honestly it doesn't matter much to keep up to date +> with this. It's an unfortunate fact that a lot of +> hardware/firmware manufacturers are lazy. +> - IPMI cert is self-signed which is an absolute bitch to add an +> exception for, especially in Java. Let's Encrypt is free. +> - I filed a ticket (<a>#824963</a>) requesting a secondary user +> for IPMI. My intention was to have myself as admin (as many +> capabilities assigned to my user as necessary to allow me to +> manage secondary, lesser users) so that I could safely give out +> controlled access as necessary to others I entrust to my infra +> (and who depend on the server as much as I do). +> - As I mentioned, Dell urges IPMI to stay on its own management +> network. I know this puts Mivo in a tough spot because there's a +> tradeoff between security and accessibility. One option would be to +> tie IPs logged-in to clients.mivocloud.com to be able to access +> IPMI for a set time. This is similar to how providers allow native +> VNC access: another provider I use generates an access IP, a +> password, and tells me I can only connect via the IP I'm signed in +> with. I usually use Tor for accessing the Web (and some customers +> may have highly-dynamic IP addresses) so this isn't a perfect +> solution, but it's a good first step. +> 6. Again unrelated, but speaking of Tor / dynamic IPs, it's difficult to +> navigate clients.mivocloud.com and stay logged-in because of how +> sessions are tied to IP addresses. It would be nice to find a +> solution to this (also to improve account security, offer TOTP +> two-factor authentication). +> +> I can create sub-issues like I said; I just wanted to summarise in one +> place first. You can close this issue when the IPv4 issue is fixed and +> either you or I can open new tickets addressing the other issues. +> +> I'm also posting my ticket on my personal blog (another reason to have +> these issues summarised in one place) and that's part of the reason I'm +> asking to consider all of these issues and be able to share how Mivo +> wants to improve from some of these issues. Let me know if I can quote +> your reply publicly to my blog. Think of it as free publicity for the +> business, I suppose ;) and yes, while I'm addressing Mivo's issues +> publicly, I'm not looking to shame anyone. I do think it's pissy that +> this stuff can't have been done right in the first place, but my intent +> here is just to ensure providers' future success and have good hosts to +> recommend to friends. +> +> Thanks + +[MivoCloud]: <https://mivocloud.com/> +[control panel]: <https://clients.mivocloud.com/clientarea.php?action=productdetails&id=80855> +[Volatile]: <https://volatile.bz/> +[iDRAC 6]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-25809/Dell-Idrac6-Firmware.html> +[Dell CVEs]: <https://www.dell.com/support/article/en-us/sln315190/dell-emc-idrac-multiple-vulnerabilities-cve-2018-15774-and-cve-2018-15776> +[iDRAC 7]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-26149/Dell-Idrac7-Firmware.html> +[iDRAC 8]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-35106/Dell-Idrac8-Firmware.html> +[iDRAC 9]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-53795/Dell-Idrac9-Firmware.html> + +---- + +like I suggested in the ticket, I'm not upset with MivoCloud, but this +isn't currently a service I can recommend in good confidence to people +looking for a hosting provider. MivoCloud has uptime, reliability, +support responsiveness, and decent price for the hardware. I've been +using their services for a few years now and have no intention of +leaving as long as these problems above are addressed, and I'd love to +recommend them to other people as well. MivoCloud just needs work on +basic practices as well as communicating with customers—for example, +outgoing port 25 traffic became blocked by default at some point, with +no warning that it was going to happen. so, suddenly I couldn't send +mails, and the firewall they use to deny outgoing smtp traffic behaved +oddly, downgrading STARTTLS connections and sending clients falsified +errors. based on that alone I thought I was being MITMed, but turns out +this was intentional by MivoCloud. and this was *after* I had told +MivoCloud support before that I was running a mailserver, so it seems +that for now, their internal management is pretty subpar and +unorganised. again, I hope it all changes for the better, and I'll +publish any updates as they come my way.