avatar

opal

wowaname

my experience with MivoCloud

I just filed an issue today with MivoCloud after IPv4 connectivity issues on my dedi. after debugging extensively, racking my brain, wasting plenty of time, and consulting with my friend about it, I realised that a subset of my /29 was pulled from under my nose and reallocated to what I suppose is another customer. the ticket:

IPv4 addresses reassigned? (and other grievances)

Hi,

This time I'm sure the issue is outside of my control. Five of the eight IPv4 addresses I pay for in my 185.163.46.96/29 block are no longer routed to my server. I verified with nmap/tcpdump as below:

~ wowaname@mahin> doas nmap -sn -PE 185.163.46.138 185.163.46.96/29
doas (wowaname@mahin) password:
Starting Nmap 7.70 ( https://nmap.org ) at 2020-03-30 06:05 -00
Nmap scan report for volatile.bz (185.163.46.138)
Host is up (0.38s latency).
Nmap scan report for 185-163-46-96.volatile.bz (185.163.46.96)
Host is up (0.38s latency).
Nmap scan report for mx1.volatile.bz (185.163.46.97)
Host is up (0.37s latency).
Nmap scan report for 185-163-46-101.volatile.bz (185.163.46.101)
Host is up (0.38s latency).
Nmap scan report for 185-163-46-102.volatile.bz (185.163.46.102)
Host is up (0.38s latency).
Nmap scan report for 185-163-46-103.volatile.bz (185.163.46.103)
Host is up (0.37s latency).
Nmap done: 9 IP addresses (6 hosts up) scanned in 1.74 seconds

and on server

# tcpdump -n -i bond0 '(icmp[icmptype]=icmp-echo or icmp[icmptype]=icmp-echoreply) and (dst 185.163.46.138 or dst net 185.163.46.96/29)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:05:31.315509 IP [home IP] > 185.163.46.102: ICMP echo request, id 20075, seq 55153, length 8
06:05:31.315726 IP [home IP] > 185.163.46.138: ICMP echo request, id 59515, seq 55157, length 8
06:05:31.315772 IP [home IP] > 185.163.46.103: ICMP echo request, id 6724, seq 55155, length 8
06:05:31.319262 IP [home IP] > 185.163.46.101: ICMP echo request, id 4617, seq 55150, length 8
^C
4 packets captured
7 packets received by filter
0 packets dropped by kernel
1 packet dropped by interface

The address http://185.163.46.96/ points to something I definitely don't host, and traceroutes differ between 185.163.46.101 (which still routes to me) and 185.163.46.96 (which routes to someone else now):

16  ae0.cr3.ias.ro.mivocloud.com (185.163.44.121)  167.040 ms  196.535 ms  204.169 ms
17  ae1.cr1.kiv.md.mivocloud.com (185.163.44.122)  238.265 ms  252.588 ms  256.559 ms
18  volatile.bz (185.163.46.138)  256.777 ms  258.746 ms  269.423 ms
19  185-163-46-101.volatile.bz (185.163.46.101)  263.569 ms  267.788 ms  271.269 ms

vs

16  ae0.cr3.ias.ro.mivocloud.com (185.163.44.121)  211.082 ms  211.650 ms  197.482 ms
17  ae1.cr1.kiv.md.mivocloud.com (185.163.44.122)  199.182 ms  198.986 ms  198.965 ms
18  185-163-46-96.volatile.bz (185.163.46.96)  198.910 ms  197.386 ms  198.901 ms

So, I'm not going to lie, I'm kind of pissed. Mistakes happen, sure, but this was entirely preventable by tracking which IPs are already allocated to users. I pay for the /29 range because I do have a need for IPv4 addresses. I wish we could go IPv6-only sooner rather than later, but that isn't reality, so I do still need IPv4, and I can't get by with NATting all connections originating from my box.

I did verify whether there was a potential for BGP hijacking but ARIN says Mivo's routing advertisements are signed using RPKI so I can safely rule that out. I've also verified that the issues I experience are not related to my home ISP: my friend has verified these issues I speak of, and random Tor exits seem to reflect my findings (e.g. accessing http://185.163.46.96/ with Tor and without).

Some questions I'll pose, and I would greatly appreciate full answers, as sysadmin to sysadmin, rather than as ISP to customer. I know I'm a customer but I go out of my way to pay for IaaS rather than anything managed, so I would rather know the infrastructure I'm working with.

  1. Ranges for "Assigned IPs" on castellum clearly indicate 185.163.46.96/29 and 2001:67c:2db8:301::/64 on the control panel. So, Mivo does have capability to track allocated IPs. What happened here so that those IPs seemed free to reuse for another customer?
  2. Mivo's networking configuration has bothered me since day one. I've been willing to work with what I was given because otherwise Mivo provides a great deal with respect for alternate payment methods (cryptocurrency is the cash of the Internet moving forward, I believe, so I use it whenever I am able). But at some point I have to say enough's enough and ask Mivo to do their part in having acceptable networking to work with. So, could we work together to figure some of this out and save pain for the future? See the next question or so for details.
  3. IPv4, I understand it's limited. That's why I chose pointopoint routing for the /29. I have a few pointed to VMs, containers, and physical machines over VPN. The routing is contrived for them though; I'd rather my gateway for these be directly to 185.163.46.129 (your gw) rather than to 172.22.3.1 and configure bird as so: route 185.163.46.97/32 via 172.22.3.101; route 185.163.46.99/32 via 172.22.3.50; route 185.163.46.101/32 via 172.22.3.60; – it wasn't too out of my way, I already had bird to participate in a shared VPN, but I still would rather my networking be more "correct". And I believe it's a good tradeoff; I have solely a /29 and I surrender a /32 back into Mivo's pool, makes it easier for all of us I believe. We can open a new ticket to track this and make sure we're prepared before switching IP configs over, for minimum downtime/disruption.
  4. IPv6, there's no reason to be stingy about /64s. In fact I'm tempted to ask for a larger range just so I can use SLAAC properly through my network. I went with it when I first found out that Mivo doesn't give full /64s as promised on registration and purchase, and I'm glad you offered me a full /64 after asking, but I'd rather not route anything through 2001:67c:2db8:9::138/128 because it makes no sense I have one IPv6 from a /64. Just please give me 2001:67c:2db8:301::/64 routed directly to your gw so I can do away with an absolutely incorrect IPv6 setup. And like I said I'm tempted to ask for a larger range (/60 is most I'd need, to slice into sixteen /64s, good enough for current and future internal use) but I'll leave that thought on the table. I do a lot of virtual networking and plan to expand (see Volatile, basically I want to offer services such as WireGuard VPN [with transparent Tor onion proxying which needs its own IPv6 range for hostmapping], mail, &c to donors and friends, to offset other cloud services like Google), and until I have enough out-of-pocket funds to build a server and colocate in the USA, I have to work with what I'm currently using. Also, I like suggesting Mivo to friends and anyone else looking for a host, and I hate telling them that "there's a catch with IPv6" – people expect an advertised /64 to work out of the box.
  5. Unrelated to this ticket and I'll create a new one if necessary: but seriously, the current IPMI setup is bad for a few reasons:
    • Most obviously is the fact it's exposed to the Internet. IPMI firmware is a hassle to update from what I understand (IIRC requires upgrading the board itself?) thus outdated firmware is host to vulnerabilities:
      1. iDRAC 6 on CVE Details
      2. CVE-2018-15774 and CVE-2018-15776 – notice how Dell explicitly recommends against exposing IPMI to the Internet
      3. iDRAC 7, iDRAC 8, iDRAC 9 all have at least one serious vuln, so honestly it doesn't matter much to keep up to date with this. It's an unfortunate fact that a lot of hardware/firmware manufacturers are lazy.
    • IPMI cert is self-signed which is an absolute bitch to add an exception for, especially in Java. Let's Encrypt is free.
    • I filed a ticket (#824963) requesting a secondary user for IPMI. My intention was to have myself as admin (as many capabilities assigned to my user as necessary to allow me to manage secondary, lesser users) so that I could safely give out controlled access as necessary to others I entrust to my infra (and who depend on the server as much as I do).
    • As I mentioned, Dell urges IPMI to stay on its own management network. I know this puts Mivo in a tough spot because there's a tradeoff between security and accessibility. One option would be to tie IPs logged-in to clients.mivocloud.com to be able to access IPMI for a set time. This is similar to how providers allow native VNC access: another provider I use generates an access IP, a password, and tells me I can only connect via the IP I'm signed in with. I usually use Tor for accessing the Web (and some customers may have highly-dynamic IP addresses) so this isn't a perfect solution, but it's a good first step.
  6. Again unrelated, but speaking of Tor / dynamic IPs, it's difficult to navigate clients.mivocloud.com and stay logged-in because of how sessions are tied to IP addresses. It would be nice to find a solution to this (also to improve account security, offer TOTP two-factor authentication).

I can create sub-issues like I said; I just wanted to summarise in one place first. You can close this issue when the IPv4 issue is fixed and either you or I can open new tickets addressing the other issues.

I'm also posting my ticket on my personal blog (another reason to have these issues summarised in one place) and that's part of the reason I'm asking to consider all of these issues and be able to share how Mivo wants to improve from some of these issues. Let me know if I can quote your reply publicly to my blog. Think of it as free publicity for the business, I suppose ;) and yes, while I'm addressing Mivo's issues publicly, I'm not looking to shame anyone. I do think it's pissy that this stuff can't have been done right in the first place, but my intent here is just to ensure providers' future success and have good hosts to recommend to friends.

Thanks

like I suggested in the ticket, I'm not upset with MivoCloud, but this isn't currently a service I can recommend in good confidence to people looking for a hosting provider. MivoCloud has uptime, reliability, support responsiveness, and decent price for the hardware. I've been using their services for a few years now and have no intention of leaving as long as these problems above are addressed, and I'd love to recommend them to other people as well. MivoCloud just needs work on basic practices as well as communicating with customers—for example, outgoing port 25 traffic became blocked by default at some point, with no warning that it was going to happen. so, suddenly I couldn't send mails, and the firewall they use to deny outgoing smtp traffic behaved oddly, downgrading STARTTLS connections and sending clients falsified errors. based on that alone I thought I was being MITMed, but turns out this was intentional by MivoCloud. and this was after I had told MivoCloud support before that I was running a mailserver, so it seems that for now, their internal management is pretty subpar and unorganised. again, I hope it all changes for the better, and I'll publish any updates as they come my way.