Linux Performance Analysis in 60 Seconds

Condensed version of this post by Brendan Gregg @ Netflix

uptime - load averages
dmesg -T | tail - kernel errors
vmstat 1 - overall stats by time
mpstat -P ALL 1 - cpu balance
pidstat 1 - process usage
iostat -xz 1 - disk i/o
free -m - memory usage
sar -n DEV 1 - network stats
sarn -n TCP,ETCP 1 - tcp stats
top - check overview

Finding a string in many files on linux

In a directory
find . -name “*ABC*” -exec grep -H ‘XYZ’ {} +

This essentially uses find to find the file, then -exec grep will search IN all the matching files for the string you specified.


This entire post was written by Mark Driver, formerly of and all copyrights I assume belong to him. The site and article has been offline for a while but I found it so insightful I saved it. It appears that Mark currently keeps a blog or site @ He can be found on twitter as @MarkDriver

1. Science has proven that the domestication of animals has resulted in a progressive erosion of their mental facilities. The domestication of humans has led to similar effects.

2. Your life is passing in front of you, you are getting old. You are going to die, and you’re never going to be on television.

3a. Get drunk and have sex at 4 am on a Tuesday in the back of a running cop car while the pig’s in the Chinese restaurant shaking poor Johnny Wong down for Chow Mein leftovers. Call in sick to work tomorrow.

3b. Stop being afraid of everything.

4. It has been structured so that we “need” money to survive. Most options for “earning” money involve us trading a large portion of our limited lives. Our “work” usually involves meaningless repetition that runs absolutely opposite to our human nature. Not only is our “work” boring, it is enforced with strict behavioral rules and the constant fear of being “fired”. It has been rammed down our throats that this is a “reality”, and that the benefits of this planned system outweigh the negatives. It is becoming increasingly clear that this may not be the case.

5. Being cool to people gives them hope. It gives you hope too. Help anyone who needs it and pay back every favor given to you in spades. Assholes are their own punishment.

6. This is my body. I’m the one who moves it around and I’m the one who makes it do amazing things. I will take full responsibility for the actions of this body. I will not be controlled by the body of another.

7. There is no priest, no politician, no boss, no cop, and no concerned citizen who will ever convince me that they are performing functions necessary to my continued existence. They do not matter, they only impede. There are no masters and there are no gods. There are only people who demand life, and people who demand control. Whose side are you on?

8. There is nothing sexy about mass production, hamburgers, or soda pop. American consumer ‘culture’ must continue to be ignored at home and worldwide. Yankee go home, and take yer shitty food with you.

9. Sex is awesome. So is a good burrito. So is a round of drinks with friends. So is reading. So is sleeping late. You shouldn’t feel bad about constantly enjoying yourself. Misery loves company. So does boredom. Ignore the telephone.

10. Freedom is the only important thing, personal liberty the only pathway to dignity. Anything that stands in the way is the enemy, no matter what costume it wears. Stop being afraid of everything.

11. Nobody knows anything. Everyone is lying. Look out for each other.

12. Do Not Pet The Tiger. The Tiger is Sick.

13a. It is an easy power to ridicule everything with sarcasm and irony, anyone can do it. It’s much harder to give things a chance and try to understand. As smart as you are, there is always something to learn, and, conversely, always something to unlearn. Stop being afraid of everything.

13b. Lighten the fuck up.

13c. Buy a gun.

Example file reaper cron

10 1 * * * /usr/bin/find /usr/local/stuff -maxdepth 1 -mtime +30 -type d | /usr/bin/xargs -r rm -r

Quick postfix queue depth script

Or counting files in any folder(s)

for i in `ls -al /var/spool/postfix |grep '^d'|awk '{print $9}'|grep -v '\.$'`; do c=`find $i|wc -l`; echo "$c $i"; done;

Gary calls in sick…

Gary: here’s a great trick for calling in sick. jami – pretend that you’re not listening. nobody copy and paste to HR. Call your boss while lying on your back with your neck hanging over the bed.
Peter: Good advice. I always just tried to sound congested, but that sounds more grounded in science.
Gary: the entire physical discomfort of the position alters your voice in such a way that you sound different from your usual self. done correctly, you’ll sound sick or you might even graduate to sounding like you’re on your deathbed.
Josh: wow. Sounds effective
Gary: that’s the only thing my dad ever taught me that should be shared, and yes: this could qualify as sharing knowledge with other teams if i send it the Office.
Sean: you could write it up in a wiki doc
Gary: in this day and age where we type instead of calling the boss, i find that it’s still an effective technique. the physical discomfort leads to typing mistakes, which makes you seem either way out of it or just plain stupid.
Sean: place the laptop on the floor. lean backwards over the bed, reach over your head and attempt to type
Gary: i just tried it. i think i broke my neck.
Sean: … the result is a valid excuse to not come into work. I think that accomplishes exactly what we were going for
Gary: +1

FreeIPA Server/Client setup on CentOS 6.5

So I’ve been dorking around with 389-ds a LOT at work lately and it’s a bitch to setup, especially when it comes to the certs.  As a hackathon project I decided to setup FreeIPA, which is the Free version of Redhat Identity Manager as a more comprehensive and easy to manage solution.  I have this setup at home as well in my personal lab.  Some pre-requisites first… make sure you have DNS and REVERSE LOOKUPS for all servers and clients and if you’re running iptables or a firewall on your hosts then make sure you have the following ports open, TCP/UDP: 888/444 for kerberos and 389/636 for ldap.

Here are the following specifics for our setup:

  • Domain:                   
  • Realm:                               EXAMPLE.COM
  • Server1:                   
  • Server2(replica):       
  • Client:                      

FreeIPA Server Setup

This setup is stupid easy, you just have yum install the ipa-server, then set it up with ipa-server-install.  Note you can just call run #ipa-server-install without any flags and it will ask you realm, domain, etc.  It will automatically setup your certificates, kerberos, etc… etc…

# yum -y install ipa-server
# ipa-server-install --realm=EXAMPLE.COM

Next let’s see if IPA is working correctly by requesting a ticket for the admin user

#kinit admin

There shouldn’t be any output, let’s validate that the ticket was issued

Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin@EXAMPLE.COM

Valid starting     Expires            Service principal
02/12/14 16:09:06  02/13/14 16:09:03  krbtgt/EXAMPLE.COM@EXAMPLE.COM
02/12/14 16:49:46  02/13/14 16:09:03  host/

Replication Setup

On the master(Server1):


Copy the gpg file that was just created at /var/lib/ipa to Server2(the replica) and run the following command on Server2:

#ipa-replica-install /var/lib/ipa/

Client Setup

I’ll specify that I find it rather important to specify the –mkhomedir flag. If you don’t then setting it up later can be a bitch.

#ipa-client-install --realm=EXAMPLE.COM -p admin  --password=<password> --mkhomedir

Enable WebUI access from anywhere

By default the WebUI is only accessible from authenticated IPA clients. This means that non-ipa clients can not access the WebUI to manage FreeIPA. While the extra security is nice, in a dev or lab setting it might be overkill. Here’s the workaround I found so you can access the webui from any computer. How the workaround actually works is that enables kerberos authentication through the web browser itself. Note that you have to do this on all freeipa servers.

On the server(s) open the ipa.conf file used by the Apache web service.

#vim /etc/httpd/conf.d/ipa.conf

In the <Location “/ipa”> location definition, change the KrbMethodK5Passwd attribute from off to on.

KrbMethodK5Passwd on

Restart the httpd service:

# service httpd restart

One small issue I found is the default admin user doesn’t seem to work if you try to access from a non-ipa client. I had to create another user(webadmin) give it admin privileges and log in to a machine as that user before I could access the WebUI from non-ipa clients.


Yum and kernels, removing old one’s and limiting how many kernels yum keeps around

Check installed kernels:
# rpm -q kernel

Remove old kernels:
# package-cleanup –oldkernels –count=2

Make it permanent:
#vi etc/yum.conf

Working with volume groups that have the same name (cloned disk, recoveries, etc)

First, we attach the vmdk to the vm and then scan the bus to see it in linux:
# echo “- – -” > /sys/class/scsi_host/host0/scan

In this scenario, /dev/sda2 and /dev/sdb2 have the same volume group name of VolGroup00, let’s rename /dev/sdb2 to VolGroup01:
# vgimportclone –basevgname VolGroup01 /dev/sdb2

next let’s find the new VolumeGroup:
# vgscan
# vgchange -a y

Now let’s mount the LV we want from the new volume group:
# mount /dev/VolGroup01/LogVol02 /mnt/restore/

Once we’re done, we want to unmount the LV and remove reference to the VG
# umount /mnt/restore
# vgchange -an VolGroup01 (make VG unavailable)
# sync

Remove the drive, wait a few minutes and let’s do a rescan:
# vgscan

Finding biggest directories in linux

find . -type d -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}