Table of Contents

• The One True Guide
• Why (are we doing this)
• How (do we flash the cards)
  • Pre-Requisites
• What (supported cards)
  • LSI Compatibility Matrix
• Troubleshooting
• Glossary of Terms

The One True Guide

See https://fohdeesha.com/docs/perc.html.

(this article is just a summary of that, as a quick reference for how we set things up)

Why are we flashing these?

Who cares! Skip ahead!

We want to run a CephFS cluster on commodity hardware.

For that we need direct, non-abstract (meaning HBA/JBOD/IT Mode) access to the drives.

There are tons of used Dell PowerEdge Rx20 series servers that are well-worn, but otherwise in good condition that are being pulled from other data centers and dumped on the market, but they don’t have good HBA

Also, this allows us to experiment with features of other modern file systems (including ZFS and BTRFS) which use the low-level access for their built-in redundancy and performance features.

The de-branded LSI firmware doesn’t have all the iDRAC management controls, but it’s necessary to make these boards usable for this purpose.

Caveats

• This guide is specific to the H710p Mini on an R620/R720/xd, no fluff
  (that said, it’s the same procedure for the similar cards)
• The LSI firmware won’t be visible to the iDRAC

How To Flash Hx10 Cards to IT Mode

Essentially you download to custom-made images and run some custom commands.

0. Download perc-crossflash-v2.3.zip
   https://fohdeesha.com/docs/store/perc/perc-crossflash-v2.3.zip
   It has everything you need. At the time of this writing:

   Version: v2.3
   ZIP Updated: 01-12-2023
   MD5: d0f15f46112762f16fe32832cd9a7492
   

   PERC Mini Crossflashing/
   ├── deesh-FreeDOS-v2.2.iso
   ├── deesh-Linux-v2.3.iso
   └── v2.3.txt
   

1. Mandatory: Remove the Battery from the PERC board (unclips easily)
2. Mandatory: Physically Remove / Disconnect all drives
3. Mandatory BIOS Settings
   • F2 to boot into the BIOS (System Setup)
     • Disable: Processor Settings > Virtualization Technology
     • Disable (default): Integrated Devices > SR-IOV Global
     • Disable (default): Integrated Devices > I/OAT DMA Engine
     • BIOS (default): Boot Settings > Boot Mode > BIOS (NOT UEFI)
4. Recommended Speed up Boot Times!
   (there are several reboots as part of this process, which account for most of the time)
   • Remove all but one stick of RAM per CPU
     (there are two RAM configurations per reboot, and they’re slow)
   • F2 on Boot for iDRAC Settings, Lifecycle Controller, Disable
   • System BIOS Settings, Miscellaneous Settings, In-System Characterization: Disabled
5. FreeDOS Time
   • connect deesh-FreeDOS-v2.2.iso as Virtual Media
   • click to Map
   • Hit F11 during boot to select Virtual CD
6. Once booted run info
   Write Down the SAS Address

   info
   

   DOS/32A -- Protected Mode Run-time Version 7.2
   Copyright (C) Supernar Systems, Ltd. 1996-2002
   Product Name : PERC H710 Mini
   ChipRevision : D1
   SAS Address :5848f690ec066900   <==== 🎗️ WRITE THIS DOWN
   

7. Based on the Product Name and Chip Revision you will need 2 custom commands
   Clear (FreeDOS)
   Flash (Linux)
   and Revert (Linux), if you ever need it
8. Run the clear command - for us, on H710/H710p Minis, it will be one of these:

   B0CROSS  # H710 Mini B0
   D1CROSS  # H710 Mini D1
   PB0CROSS # H710p Mini B0
   PD1CROSS # H710p Mini D1
   

9. Linux time:
   • reboot to reboot FreeDOS
   • Connect and Map deesh-Linux-v2.3.iso
   • it’s normal to get a kernel panic on the first reboot just reboot again
10. Switch to the root user:

    sudo su -
    

11. Run the flash command - for us that will be one of these:

    B0-H710  # all B0 variants of H710/H710p/H810 cards
    D1-H710  # all D1 variants of H710/H710p/H810 cards
    H310     # all H310 variants
    

    And reboot:

    shutdown -r now
    

12. Boot errors are normal for the first 1 or 2 boots.
    It’s just the various firmware systems being out of sync.
    If you encouter core, cache, kernel or not found errors, just reboot and retry.
13. Now set the SAS Address and add boot capabilities (if you boot that way):

    # set the SAS Address
    setsas 0000000-0-0000-0000
    
    # optionally add support for BIOS boot
    flashboot ./Bootloaders/mptsas2.rom
    
    # optionally add support for UEFI boot
    flashboot ./Bootloaders/x64sas2.rom
    
    # optionally turn off 3rd party fan failsafe
    # (will rely only on other temps because the iDRAC won't get HDD temp anymore)
    ipmitool –I open raw 0x30 0xce 0 0x16 5 0 0 0 5 0 1 0 0
    

    Optionally, you can run info to check the card info:

    info
    

        Controller Number              : 0
        Controller                     : SAS2308_2(D1)
        PCI Address                    : 00:02:00:00
        SAS Address                    : 0000000-0-0000-0000
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2214 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9207-8i
        BIOS Version                   : N/A
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS9207-8i
        Board Assembly                 : N/A
        Board Tracer Number            : N/A
    

14. Reboot and put your BIOS settings back to normal:

• F2 to boot into the BIOS (System Setup)
  • Enable: Processor Settings > Virtualization Technology
  • Enable: Lifecycle Controller

Done! Now it’s ready to use!

Pre-Req

BIOS v2.9.0

We’re using the v2.9.0 BIOS.

• R620 BIOS_8P8WX_WN64_2.9.0.EXE (exe)
• R720/R720xd BIOS_W1WJW_WN64_2.9.0.EXE (exe)

And we do all this remotely via the v2.65.65.65 iDRAC.

iDRAC v2.65.65.65

• Version 2.65.65.65 (Build 15) (iDRAC-with-Lifecycle-Controller_Firmware_0GHF4_WN64_2.65.65.65_A00.EXE exe)
  • The .EXE is the upload format for the iDRAC. There is no “plain zip”.
• Default Secure Web Admin: https://192.168.0.120
• Username: root
• Password: calvin
• Overview > Server > Virtual Console > Plug-in Type > HTML5
  • Launch Virtual Console > Connect Virtual Media

note: the .BIN files aren’t actually firmware “bin"s, they’re self-extracting shell scripts. They WILL NOT WORK for BIOS or iDRAC updates.

Supported Cards

These are pretty much the same LSI-manufactured chips, in different packages with different amounts of RAM, etc:

• H310 / H710 / H710p / H810
  • Mini Mono, Blade, and Full Size PCIe variants

LSI Compatibility Matrix

From the FreeDOS .iso you can run an info command that gives product & revision info about your card:

info

Product Name : PERC H710 Mini
ChipRevision : D1

Using that information, you’ll be able to find which (ALL-CAPS) commands to use here:

Troubleshooting

If you have any troubles or curiosities, consult The One True Guide for extra details.

Remember, these things are only $35 on ebay, so it’s not a big deal if you brick one - it’s probably a better use of your time just to buy them.

FYI: The “Mini Mono"s are considered “Integrated RAID” in the BIOS. Yes, they are a card but… it’s a daughterboard, they’re still IR.

Glossary of Terms

Art of Server: RAID vs HBA SAS Controllers