The Geek Lab

1275 Members
A place to discuss Linux and open source related technologies. Please keep conversations family friendly211 Servers

Load older messages

26 Jun 2022
@telegram_763555945:t2bot.ioDeleted Account @q5sys hi, I'm switching to a 100G and I have a question about the mellanox connectx-3 can you switch them to a 40g Eth or its 40g IB and 10G eth ??? 13:21:49
@telegram_763555945:t2bot.ioDeleted AccountI find different information everywhere and I don't know what to trust13:22:01
@telegram_72424719:t2bot.ioJT (q5sys)
In reply to Deleted Account
@q5sys hi, I'm switching to a 100G and I have a question about the mellanox connectx-3 can you switch them to a 40g Eth or its 40g IB and 10G eth ???
connectx-3 cards that are 56GB IB will do 40GB ETH... IF they are dual mode cards.
@telegram_72424719:t2bot.ioJT (q5sys)some connectx-3 cards that do 56GB IB are Infiniband only.13:25:43
@telegram_72424719:t2bot.ioJT (q5sys)the connectx-2 cards that were 40gb IB would only do 10gb ETH.13:25:56
@telegram_72424719:t2bot.ioJT (q5sys)because it's 10gb per link... and x2 cards could only use one link when in eth mode...13:26:30
@telegram_72424719:t2bot.ioJT (q5sys)that was the improvement for the connectx3 generation... that and the bump from 10gb to 14gb per link13:26:48
@telegram_72424719:t2bot.ioJT (q5sys) https://github.com/q5sys/talks/blob/master/self-2022-06/slide-20.gif 13:28:01
@telegram_72424719:t2bot.ioJT (q5sys)That was using MCX354A-FCBT cards13:29:23
@telegram_72424719:t2bot.ioJT (q5sys) If you're looking for cards... look for cards that are VPI 13:30:11
@telegram_72424719:t2bot.ioJT (q5sys) * If you're looking for Mellanox IB cards... look for cards that are VPI those will be the ones that will also do ETH. 13:30:34
@telegram_72424719:t2bot.ioJT (q5sys)Dont trust a listing on ebay to get the specs right... find the part number of the card and look up the specs on mellanox's site.13:31:11
@telegram_72424719:t2bot.ioJT (q5sys)and then, when you get the card, verify its the exact card model that was listed.13:32:09
@telegram_72424719:t2bot.ioJT (q5sys)A lot of the liquidation places do so much inventory, that they'll sometimes list inventory on sites under the wrong model.13:33:04
@telegram_72424719:t2bot.ioJT (q5sys)Pro-tip... when you get the box... record a video of you opening the box and getting the model number. It can be a handheld phone video, just something so if they try to dispute saying "we sent the right card you're trying to scam us", you have proof of the exact card you bought. I've never had ebay rule against me when I submit a video of me opening the package.13:34:33
@telegram_763555945:t2bot.ioDeleted AccountI never had anybody dispute that I didn't get the correct item :D13:40:40
@telegram_763555945:t2bot.ioDeleted Accountbut thx for the info13:40:50
@telegram_763555945:t2bot.ioDeleted AccountI got qlogic ib cards because of the intel fabric switch, they were compatible, but now I'll ditch IB in general and move to eth only13:42:01
@telegram_763555945:t2bot.ioDeleted Accountceph doesn't support IB, gluster doesn't support IB so unless I want to go for vmware there is no need for IB as a linux user13:42:35
@telegram_72424719:t2bot.ioJT (q5sys)IB is nice to have, but it really only makes a difference if you're A) running a cluster or B) really need RDMA.13:42:48
@telegram_72424719:t2bot.ioJT (q5sys)and by cluster I mean compute cluster... not storage.13:43:16
@telegram_763555945:t2bot.ioDeleted Accountyeah I was doing HCI with ovirt + gluster the problem was that IB wasn't working so no RDMA and I got only IPoIB13:44:07
@telegram_72424719:t2bot.ioJT (q5sys)and ipoib gimps your bandwidth13:44:37
@telegram_763555945:t2bot.ioDeleted Accountredhat dropped support for rdma from rhel8.313:44:39
@telegram_72424719:t2bot.ioJT (q5sys)yea that struck me as odd, thankfully its still possible to use... but its unsupported.13:45:06
@telegram_72424719:t2bot.ioJT (q5sys)but everyone that I know that is using it... probably has enough of a $ sized contract, that RH will still support them on the down low.13:45:47
@telegram_763555945:t2bot.ioDeleted Accountits been such a dumpster fire with the whole IB I've decided to ditch it as a whole13:45:56
@telegram_763555945:t2bot.ioDeleted Account500 euro for the initial IB setup and now another 1000 for the 40G eth13:47:43
@esc_byte:matrix.orgesc_byte joined the room.17:55:40
@lunax:matrix.orglunax joined the room.23:19:59

There are no newer messages yet.

Back to Room List