| 26 Jun 2022 |
 Deleted Account | @q5sys hi, I'm switching to a 100G and I have a question about the mellanox connectx-3 can you switch them to a 40g Eth or its 40g IB and 10G eth ??? | 13:21:49 |
| Deleted Account | I find different information everywhere and I don't know what to trust | 13:22:01 |
 JT (q5sys) | In reply to Deleted Account
@q5sys hi, I'm switching to a 100G and I have a question about the mellanox connectx-3 can you switch them to a 40g Eth or its 40g IB and 10G eth ??? connectx-3 cards that are 56GB IB will do 40GB ETH... IF they are dual mode cards. | 13:25:27 |
| JT (q5sys) | some connectx-3 cards that do 56GB IB are Infiniband only. | 13:25:43 |
| JT (q5sys) | the connectx-2 cards that were 40gb IB would only do 10gb ETH. | 13:25:56 |
| JT (q5sys) | because it's 10gb per link... and x2 cards could only use one link when in eth mode... | 13:26:30 |
| JT (q5sys) | that was the improvement for the connectx3 generation... that and the bump from 10gb to 14gb per link | 13:26:48 |
| JT (q5sys) | https://github.com/q5sys/talks/blob/master/self-2022-06/slide-20.gif | 13:28:01 |
| JT (q5sys) | That was using MCX354A-FCBT cards | 13:29:23 |
| JT (q5sys) | If you're looking for cards... look for cards that are VPI | 13:30:11 |
| JT (q5sys) | * If you're looking for Mellanox IB cards... look for cards that are VPI those will be the ones that will also do ETH. | 13:30:34 |
| JT (q5sys) | Dont trust a listing on ebay to get the specs right... find the part number of the card and look up the specs on mellanox's site. | 13:31:11 |
| JT (q5sys) | and then, when you get the card, verify its the exact card model that was listed. | 13:32:09 |
| JT (q5sys) | A lot of the liquidation places do so much inventory, that they'll sometimes list inventory on sites under the wrong model. | 13:33:04 |
| JT (q5sys) | Pro-tip... when you get the box... record a video of you opening the box and getting the model number. It can be a handheld phone video, just something so if they try to dispute saying "we sent the right card you're trying to scam us", you have proof of the exact card you bought. I've never had ebay rule against me when I submit a video of me opening the package. | 13:34:33 |
| Deleted Account | I never had anybody dispute that I didn't get the correct item :D | 13:40:40 |
| Deleted Account | but thx for the info | 13:40:50 |
| Deleted Account | I got qlogic ib cards because of the intel fabric switch, they were compatible, but now I'll ditch IB in general and move to eth only | 13:42:01 |
| Deleted Account | ceph doesn't support IB, gluster doesn't support IB so unless I want to go for vmware there is no need for IB as a linux user | 13:42:35 |
| JT (q5sys) | IB is nice to have, but it really only makes a difference if you're A) running a cluster or B) really need RDMA. | 13:42:48 |
| JT (q5sys) | and by cluster I mean compute cluster... not storage. | 13:43:16 |
| Deleted Account | yeah I was doing HCI with ovirt + gluster the problem was that IB wasn't working so no RDMA and I got only IPoIB | 13:44:07 |
| JT (q5sys) | and ipoib gimps your bandwidth | 13:44:37 |
| Deleted Account | redhat dropped support for rdma from rhel8.3 | 13:44:39 |
| JT (q5sys) | yea that struck me as odd, thankfully its still possible to use... but its unsupported. | 13:45:06 |
| JT (q5sys) | but everyone that I know that is using it... probably has enough of a $ sized contract, that RH will still support them on the down low. | 13:45:47 |
| Deleted Account | its been such a dumpster fire with the whole IB I've decided to ditch it as a whole | 13:45:56 |
| Deleted Account | 500 euro for the initial IB setup and now another 1000 for the 40G eth | 13:47:43 |
|  esc_byte joined the room. | 17:55:40 |
|  lunax joined the room. | 23:19:59 |
