Wednesday, November 9, 2011

RHEL6 latency on Dell M610: C States

We ran into latency problems on Dell M610 servers using RHEL6 while RHEL5 performed perfectly. The RHEL6 stock kernel has a module called intel_idle which ignores Dell's BIOS settings to disable C-states. We've set the following in the tail of the boot line in /etc/grub.conf and so far so good.

intel_idle.max_cstate=0 processor.max_cstate=1
When I started here six years ago we had a DNS server with a similar problem solved basically the same way; with a "acpi=off" to disable ACPI.

Update: The above seems to have helped servers connected to the SAN but not one which just uses the local disk. RedHat had some interesting tweaks to try with MegaCli, hdparm, and the scheduler as they think it's IO related:

Set policy to cache data:

# MegaCli64 -LDSetProp -Cached -LAll -aAll
Enable disks' cache:
# MegaCli64 -LDSetProp EnDskCache -LAll -aAll
# MegaCli64 -LDSetProp ADRA -LALL -aALL
Enable write cache, but first verify if the controller is battery backed:
# MegaCli64 -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood' -e '^Charger Status' 
If the above results display no BBU, do not proceed.
If we have a battery backed controller, enable write cache.
# MegaCli64 -LDSetProp WB -LALL -aALL
If battery fails or is discharged, disable write cache:
# MegaCli64 -LDSetProp NoCachedBadBBU -LALL -aALL
Verify policy is Enabled:
# MegaCli64 -LDInfo -LAll -aAll |grep 'Disk Cache'
Disk Cache Policy: Enabled
Disable local device caching:
# hdparm -W0 /dev/sda
- Add the above to /etc/rc.local to persist reboots.
Edit /etc/fstab and modify the mount options, ie: 
/dev/mapper/vg_root-lv_root / ext4 defaults,nobarriers,data=writeback 1 1
...
...
The filesystem(s) would need be remounted for changes to take effect.
Next, we will change the scheduler to Deadline.
# echo 'deadline' > /sys/block/sda/queue/scheduler
# echo 200 > /sys/block/sda/queue/iosched/read_expire
# echo 500 > /sys/block/sda/queue/iosched/write_expire
- Add the above commands to /etc/rc.local to persist reboots.