wowana.me

website source; use git clone git://wowana.me/wowana.me.git to clone this repository.


my-experience-with-mivocloud.md (12958B)


      1 # my experience with MivoCloud
      2 <!--[time 202003300802.17]-->
      3 
      4 I just filed an issue today with [MivoCloud] [] after IPv4 connectivity
      5 issues on my dedi. after debugging extensively, racking my brain,
      6 wasting plenty of time, and consulting with my friend about it, I
      7 realised that a subset of my /29 was pulled from under my nose and
      8 reallocated to what I suppose is another customer. the ticket:
      9 
     10 > ## IPv4 addresses reassigned? (and other grievances)
     11 > 
     12 > Hi,
     13 > 
     14 > This time I'm sure the issue is outside of my control. Five of the eight
     15 > IPv4 addresses I pay for in my 185.163.46.96/29 block are no longer
     16 > routed to my server. I verified with nmap/tcpdump as below:
     17 > 
     18 >     ~ wowaname@mahin> doas nmap -sn -PE 185.163.46.138 185.163.46.96/29
     19 >     doas (wowaname@mahin) password:
     20 >     Starting Nmap 7.70 ( https://nmap.org ) at 2020-03-30 06:05 -00
     21 >     Nmap scan report for volatile.bz (185.163.46.138)
     22 >     Host is up (0.38s latency).
     23 >     Nmap scan report for 185-163-46-96.volatile.bz (185.163.46.96)
     24 >     Host is up (0.38s latency).
     25 >     Nmap scan report for mx1.volatile.bz (185.163.46.97)
     26 >     Host is up (0.37s latency).
     27 >     Nmap scan report for 185-163-46-101.volatile.bz (185.163.46.101)
     28 >     Host is up (0.38s latency).
     29 >     Nmap scan report for 185-163-46-102.volatile.bz (185.163.46.102)
     30 >     Host is up (0.38s latency).
     31 >     Nmap scan report for 185-163-46-103.volatile.bz (185.163.46.103)
     32 >     Host is up (0.37s latency).
     33 >     Nmap done: 9 IP addresses (6 hosts up) scanned in 1.74 seconds
     34 > 
     35 > and on server
     36 > 
     37 >     # tcpdump -n -i bond0 '(icmp[icmptype]=icmp-echo or icmp[icmptype]=icmp-echoreply) and (dst 185.163.46.138 or dst net 185.163.46.96/29)'
     38 >     tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
     39 >     listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
     40 >     06:05:31.315509 IP [home IP] > 185.163.46.102: ICMP echo request, id 20075, seq 55153, length 8
     41 >     06:05:31.315726 IP [home IP] > 185.163.46.138: ICMP echo request, id 59515, seq 55157, length 8
     42 >     06:05:31.315772 IP [home IP] > 185.163.46.103: ICMP echo request, id 6724, seq 55155, length 8
     43 >     06:05:31.319262 IP [home IP] > 185.163.46.101: ICMP echo request, id 4617, seq 55150, length 8
     44 >     ^C
     45 >     4 packets captured
     46 >     7 packets received by filter
     47 >     0 packets dropped by kernel
     48 >     1 packet dropped by interface
     49 > 
     50 > The address <http://185.163.46.96/> points to something I definitely
     51 > don't host, and traceroutes differ between 185.163.46.101 (which still
     52 > routes to me) and 185.163.46.96 (which routes to someone else now):
     53 > 
     54 >     16  ae0.cr3.ias.ro.mivocloud.com (185.163.44.121)  167.040 ms  196.535 ms  204.169 ms
     55 >     17  ae1.cr1.kiv.md.mivocloud.com (185.163.44.122)  238.265 ms  252.588 ms  256.559 ms
     56 >     18  volatile.bz (185.163.46.138)  256.777 ms  258.746 ms  269.423 ms
     57 >     19  185-163-46-101.volatile.bz (185.163.46.101)  263.569 ms  267.788 ms  271.269 ms
     58 > 
     59 > vs
     60 > 
     61 >     16  ae0.cr3.ias.ro.mivocloud.com (185.163.44.121)  211.082 ms  211.650 ms  197.482 ms
     62 >     17  ae1.cr1.kiv.md.mivocloud.com (185.163.44.122)  199.182 ms  198.986 ms  198.965 ms
     63 >     18  185-163-46-96.volatile.bz (185.163.46.96)  198.910 ms  197.386 ms  198.901 ms
     64 > 
     65 > So, I'm not going to lie, I'm kind of pissed. Mistakes happen, sure, but
     66 > this was entirely preventable by tracking which IPs are already
     67 > allocated to users. I pay for the /29 range because I do have a need for
     68 > IPv4 addresses. I wish we could go IPv6-only sooner rather than later,
     69 > but that isn't reality, so I do still need IPv4, and I can't get by with
     70 > NATting all connections originating from my box.
     71 > 
     72 > I did verify whether there was a potential for BGP hijacking but ARIN
     73 > says Mivo's routing advertisements are signed using RPKI so I can safely
     74 > rule that out. I've also verified that the issues I experience are not
     75 > related to my home ISP: my friend has verified these issues I speak of,
     76 > and random Tor exits seem to reflect my findings (e.g. accessing
     77 > <http://185.163.46.96/> with Tor and without).
     78 > 
     79 > Some questions I'll pose, and I would greatly appreciate full answers,
     80 > as sysadmin to sysadmin, rather than as ISP to customer. I know I'm a
     81 > customer but I go out of my way to pay for IaaS rather than anything
     82 > managed, so I would rather know the infrastructure I'm working with.
     83 > 
     84 > 1. Ranges for "Assigned IPs" on castellum clearly indicate
     85 >    `185.163.46.96/29` and `2001:67c:2db8:301::/64` on the [control
     86 >    panel] [].  So, Mivo does have capability to track allocated IPs.
     87 >    What happened here so that those IPs seemed free to reuse for
     88 >    another customer?
     89 > 2. Mivo's networking configuration has bothered me since day one. I've
     90 >    been willing to work with what I was given because otherwise Mivo
     91 >    provides a great deal with respect for alternate payment methods
     92 >    (cryptocurrency is the cash of the Internet moving forward, I
     93 >    believe, so I use it whenever I am able). But at some point I have to
     94 >    say enough's enough and ask Mivo to do their part in having
     95 >    acceptable networking to work with. So, could we work together to
     96 >    figure some of this out and save pain for the future? See the next
     97 >    question or so for details.
     98 > 3. IPv4, I understand it's limited. That's why I chose pointopoint
     99 >    routing for the /29. I have a few pointed to VMs, containers, and
    100 >    physical machines over VPN. The routing is contrived for them though;
    101 >    I'd rather my gateway for these be directly to 185.163.46.129 (your
    102 >    gw) rather than to 172.22.3.1 and configure bird as so: `route
    103 >    185.163.46.97/32 via 172.22.3.101; route 185.163.46.99/32 via
    104 >    172.22.3.50; route 185.163.46.101/32 via 172.22.3.60;` – it wasn't
    105 >    too out of my way, I already had bird to participate in a shared VPN,
    106 >    but I still would rather my networking be more "correct". And I
    107 >    believe it's a good tradeoff; I have solely a /29 and I surrender a
    108 >    /32 back into Mivo's pool, makes it easier for all of us I believe.
    109 >    We can open a new ticket to track this and make sure we're prepared
    110 >    before switching IP configs over, for minimum downtime/disruption.
    111 > 4. IPv6, there's *no* reason to be stingy about /64s. In fact I'm
    112 >    tempted to ask for a larger range just so I can use SLAAC properly
    113 >    through my network. I went with it when I first found out that Mivo
    114 >    doesn't give full /64s as promised on registration and purchase, and
    115 >    I'm glad you offered me a full /64 after asking, but I'd rather not
    116 >    route anything through 2001:67c:2db8:9::138/128 because it makes no
    117 >    sense I have one IPv6 from a /64. Just please give me
    118 >    2001:67c:2db8:301::/64 routed directly to your gw so I can do away
    119 >    with an absolutely incorrect IPv6 setup. And like I said I'm tempted
    120 >    to ask for a larger range (/60 is most I'd need, to slice into
    121 >    sixteen /64s, good enough for current and future internal use) but
    122 >    I'll leave that thought on the table. I do a lot of virtual
    123 >    networking and plan to expand (see [Volatile] [], basically I want
    124 >    to offer services such as WireGuard VPN [with transparent Tor onion
    125 >    proxying which needs its own IPv6 range for hostmapping], mail, &c
    126 >    to donors and friends, to offset other cloud services like Google),
    127 >    and until I have enough out-of-pocket funds to build a server and
    128 >    colocate in the USA, I have to work with what I'm currently using.
    129 >    Also, I like suggesting Mivo to friends and anyone else looking for
    130 >    a host, and I hate telling them that "there's a catch with IPv6" –
    131 >    people expect an advertised /64 to work out of the box.
    132 > 5. Unrelated to this ticket and I'll create a new one if necessary: but
    133 >    seriously, the current IPMI setup is bad for a few reasons:
    134 >    - Most obviously is the fact it's exposed to the Internet. IPMI
    135 >      firmware is a hassle to update from what I understand (IIRC
    136 >      requires upgrading the board itself?) thus outdated firmware is
    137 >      host to vulnerabilities:
    138 >      1. [iDRAC 6 on CVE Details] [iDRAC 6]
    139 >      2. [CVE-2018-15774 and CVE-2018-15776] [Dell CVEs] – notice how
    140 >         Dell explicitly recommends against exposing IPMI to the
    141 >         Internet
    142 >      3. [iDRAC 7], [iDRAC 8], [iDRAC 9] all have at least one serious
    143 >         vuln, so honestly it doesn't matter much to keep up to date
    144 >         with this. It's an unfortunate fact that a lot of
    145 >         hardware/firmware manufacturers are lazy.
    146 >    - IPMI cert is self-signed which is an absolute bitch to add an
    147 >      exception for, especially in Java. Let's Encrypt is free.
    148 >    - I filed a ticket (<a>#824963</a>) requesting a secondary user
    149 >      for IPMI. My intention was to have myself as admin (as many
    150 >      capabilities assigned to my user as necessary to allow me to
    151 >      manage secondary, lesser users) so that I could safely give out
    152 >      controlled access as necessary to others I entrust to my infra
    153 >      (and who depend on the server as much as I do).
    154 >    - As I mentioned, Dell urges IPMI to stay on its own management
    155 >      network. I know this puts Mivo in a tough spot because there's a
    156 >      tradeoff between security and accessibility. One option would be to
    157 >      tie IPs logged-in to clients.mivocloud.com to be able to access
    158 >      IPMI for a set time. This is similar to how providers allow native
    159 >      VNC access: another provider I use generates an access IP, a
    160 >      password, and tells me I can only connect via the IP I'm signed in
    161 >      with. I usually use Tor for accessing the Web (and some customers
    162 >      may have highly-dynamic IP addresses) so this isn't a perfect
    163 >      solution, but it's a good first step.
    164 > 6. Again unrelated, but speaking of Tor / dynamic IPs, it's difficult to
    165 >    navigate clients.mivocloud.com and stay logged-in because of how
    166 >    sessions are tied to IP addresses. It would be nice to find a
    167 >    solution to this (also to improve account security, offer TOTP
    168 >    two-factor authentication).
    169 > 
    170 > I can create sub-issues like I said; I just wanted to summarise in one
    171 > place first. You can close this issue when the IPv4 issue is fixed and
    172 > either you or I can open new tickets addressing the other issues.
    173 > 
    174 > I'm also posting my ticket on my personal blog (another reason to have
    175 > these issues summarised in one place) and that's part of the reason I'm
    176 > asking to consider all of these issues and be able to share how Mivo
    177 > wants to improve from some of these issues. Let me know if I can quote
    178 > your reply publicly to my blog. Think of it as free publicity for the
    179 > business, I suppose ;) and yes, while I'm addressing Mivo's issues
    180 > publicly, I'm not looking to shame anyone. I do think it's pissy that
    181 > this stuff can't have been done right in the first place, but my intent
    182 > here is just to ensure providers' future success and have good hosts to
    183 > recommend to friends.
    184 > 
    185 > Thanks
    186 
    187 [MivoCloud]: <https://mivocloud.com/>
    188 [control panel]: <https://clients.mivocloud.com/clientarea.php?action=productdetails&id=80855>
    189 [Volatile]: <https://volatile.bz/>
    190 [iDRAC 6]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-25809/Dell-Idrac6-Firmware.html>
    191 [Dell CVEs]: <https://www.dell.com/support/article/en-us/sln315190/dell-emc-idrac-multiple-vulnerabilities-cve-2018-15774-and-cve-2018-15776>
    192 [iDRAC 7]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-26149/Dell-Idrac7-Firmware.html>
    193 [iDRAC 8]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-35106/Dell-Idrac8-Firmware.html>
    194 [iDRAC 9]: <https://www.cvedetails.com/vulnerability-list/vendor_id-2234/product_id-53795/Dell-Idrac9-Firmware.html>
    195 
    196 ----
    197 
    198 like I suggested in the ticket, I'm not upset with MivoCloud, but this
    199 isn't currently a service I can recommend in good confidence to people
    200 looking for a hosting provider. MivoCloud has uptime, reliability,
    201 support responsiveness, and decent price for the hardware. I've been
    202 using their services for a few years now and have no intention of
    203 leaving as long as these problems above are addressed, and I'd love to
    204 recommend them to other people as well. MivoCloud just needs work on
    205 basic practices as well as communicating with customers—for example,
    206 outgoing port 25 traffic became blocked by default at some point, with
    207 no warning that it was going to happen. so, suddenly I couldn't send
    208 mails, and the firewall they use to deny outgoing smtp traffic behaved
    209 oddly, downgrading STARTTLS connections and sending clients falsified
    210 errors. based on that alone I thought I was being MITMed, but turns out
    211 this was intentional by MivoCloud. and this was *after* I had told
    212 MivoCloud support before that I was running a mailserver, so it seems
    213 that for now, their internal management is pretty subpar and
    214 unorganised. again, I hope it all changes for the better, and I'll
    215 publish any updates as they come my way.