Showing posts with label ZFS. Show all posts
Showing posts with label ZFS. Show all posts

Sunday, December 30, 2012

FreeBSD下用 shell script 建立ZFS snapshot 更新版

此次更新的目的是讓 ZFS 不使用 ls /PATH/.zfs/snapshot 目錄判斷快照數量(為了避開 Bad file descriptor 問題). 中間版本曾經使用 /usr/local/bin/snapshot list 進行, 但是速度較慢. 好在這個檔案是 shell script. 看了一下 op_list 函式的內容看到取得目前已有的快照方法, 也了解到這隻程式慢在還要查詢及計算使用空間等數據. 所以將查詢改用 zfs list -H -t snapshot -o name | grep "ZPOOL_NAME/ZVOL_NAME@". (ZPOOL_NAME/ZVOL_NAME 可以使用 zfs list -H -o name ${volpath} 查詢得到)

Saturday, November 17, 2012

在 FreeBSD 建立樣版功能的 jail 環境


0. 準備步驟 :
0.0. 準備 Host 的網路環境
0.0.0. 綁 IP 到 Host : 編輯 /etc/rc.conf 加上 IP alias 設定
ifconfig_em0_alias1="192.168.66.66 netmask 255.255.255.255"

0.0.1. 設定 natd (如果是只有一片網卡的機器就可以不用, 此步驟適用機器有跨在兩個網路(一內一外)上)
Edit /etc/rc.conf

gateway_enable="YES"

firewall_enable="YES"
firewall_type="/etc/my.firewall"
firewall_script="/etc/my.firewall"
firewall_logging="YES"

natd_enable="YES"
natd_interface="re0"
natd_flags="-l -f /etc/natd.conf"


Edit /etc/my.firewall (請依照實際需求更改)

#!/bin/sh -

fwcmd="/sbin/ipfw"                                                                                               
INTIF="re0"
LANIF0="em0"


${fwcmd} -f flush                                                                                              
#For NATD                                                                                                      
${fwcmd} add 00030 divert natd all from any to any via ${INTIF}
...
${fwcmd} add 65000 pass all from any to any

Edit /etc/natd.conf

use_sockets yes
same_ports yes
dynamic yes

Edit /etc/services
natd            8668/divert # Network Address Translation


接著 Host 重新啟動.

0.1. 在 /usr/src 編譯整個環境.
# cd /usr/src; make buildworld

1. 設定樣板存放位置且安裝樣板

# zfs create storage/jail
# zfs set mountpoint=/storage/jail storage/jail
# zfs set dedup=on storage/jail
# zfs set quota=300G storage/jail
# zfs create storage/jails
# zfs set mountpoint=/storage/jails storage/jails
# zfs set dedup=on storage/jails
# zfs set quota=300G storage/jails
# mkdir -p /storage/jail/mroot

# cd /usr/src
# make installworld DESTDIR=/storage/jail/mroot

2. 複製 src 及 ports 
# cd /storage/jail/mroot
# mkdir usr/ports
# portsnap -p /storage/jail/mroot/usr/ports fetch extract
# cpdup /usr/src /storage/jail/mroot/usr/src

3. 建立樣板環境
# mkdir /storage/jail/skel /storage/jail/skel/home /storage/jail/skel/usr-X11R6 /storage/jail/skel/distfiles /storage/jail/skel/packages
# mv etc /storage/jail/skel
# mv usr/local /storage/jail/skel/usr-local
# mv tmp /storage/jail/skel
# mv var /storage/jail/skel
# mv root /storage/jail/skel

4. 產生設定檔
# mergemaster -t /storage/jail/skel/var/tmp/temproot -D /storage/jail/skel -i
# cd /storage/jail/skel
# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev

5. 建立個別環境可寫入的目錄
#cd /storage/jail/mroot
#mkdir s
#ln -s s/etc etc
#ln -s s/home home
#ln -s s/root root
#ln -s ../s/usr-local usr/local
#ln -s ../s/usr-X11R6 usr/X11R6
#ln -s ../../s/distfiles usr/ports/distfiles
#ln -s ../../s/packages usr/ports/packages
#ln -s s/tmp tmp
#ln -s s/var var

6. 建立 jailed O.S. 的 make.conf
# Add /storage/jail/skel/etc/make.conf
# WRKDIRPREFIX?=  /s/portbuild

以下是每個 jailed O.S. 都要做一次

7. 設定 Host 每次開機為 jailed O.S. 掛上目錄
[FreeBSD 9 labdata2] (Why: 開機時因 ZFS 來不及掛載而無法在 /etc/fstab 處理)
Edit /etc/rc.local
### For jail
/sbin/mount -t nullfs -r /storage/jail/mroot /storage/jail/labdata2
/sbin/mount -t nullfs -rw /storage/jails/labdata2 /storage/jail/labdata2/s

[FreeBSD 8 localdata2]
Edit /etc/fstab

#/storage/jail/mroot   /storage/jail/localdata2     nullfs  ro  0   0
#/storage/jails/localdata2     /storage/jail/localdata2/s   nullfs  rw  0   0


8. 編輯 Host 的 /etc/rc.conf
###20121004
jail_enable="YES"
jail_set_hostname_allow="NO"
### 以下是清單, 有幾台 jailed O.S. 就登記幾台
jail_list="localdata2"
### 以下是每一台 jailed O.S. 的設定記得每一台都要有一組設定.(前四行是必要的)
jail_localdata2_hostname="localdata2.DOMAIN_NAME"
jail_localdata2_ip="192.168.66.66"
jail_localdata2_interface="em0"
jail_localdata2_rootdir="/storage/jail/localdata2"
jail_localdata2_devfs_enable="YES"
### 20121113 解決 netstat -rn 查詢問題
#jail_localdata2_devfs_ruleset="devfsrules_jail"
jail_localdata2_devfs_ruleset="devfsrules_jail2"
jail_localdata2_exec_start="/bin/sh /etc/rc"
jail_localdata2_exec_stop="/bin/sh /etc/rc.shutdown"
jail_localdata2_fdescfs_enable="YES"
jail_localdata2_procfs_enable="YES"

9. 建立系統目錄
mkdir /storage/jail/localdata2
mkdir /storage/jails/localdata2
cpdup /storage/jail/skel /storage/jails/localdata2
mkdir /storage/jail/localdata2/s

10. [Optional] 在 Host 編輯 /etc/sysctl.conf 開放 jailed O.S. 使用 ping.
Edit /etc/sysctl.conf
###20121113 For jail to fix ping icmp socket: Operation not permitted
security.jail.allow_raw_sockets=1

11. [Optional] 在 Host 編輯 /etc/default/devfs.rules 開放 jailed O.S. 使用 pts pty mem bpf
11.1. Edit /etc/default/devfs.rules
[devfsrules_unhide_login=3]
...
add path pts unhide
add path 'pts/*' unhide
add path pty unhide
add path 'pty/*' unhide

11.2. Edit /etc/devfs.rules
###
### http://forums.freebsd.org/archive/index.php/t-24581.html
### http://forums.freebsd.org/archive/index.php/t-5693.html
###
###

[devfsrules_unhide_mem=5]
add path mem unhide
add path kmem unhide

[devfsrules_unhide_bpf=6]
add path bpf unhide

# Devices usually found in a jail.
#
[devfsrules_jail2=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_mem
add include $devfsrules_unhide_bpf

12. 掛載目錄
[FreeBSD 9]

/sbin/mount -t nullfs -r /storage/jail/mroot /storage/jail/labdata2
/sbin/mount -t nullfs -rw /storage/jails/labdata2 /storage/jail/labdata2/s


[FreeBSD 8]
mount /storage/jail/localdata2
mount /storage/jail/localdata2/s

13. 啟動 jail
# /etc/rc.d/jail start
查詢啟動的 jailed O.S.
# jls
進入 jailed O.S.
# jexec #JID tcsh (或 jexec #JID /ur/bin/su -)

14. 編輯 jailed O.S. 的系統設定
Edit /etc/rc.conf
Edit /etc/resolv.conf
...


參考資料
1. FreeBSD 使用手冊  15.6 Jail 的應用
2. Forums.freebsd.org :  [Solved] jails with vnet in rc.conf
3. Forums.freebsd.org : jail nat route

Saturday, July 28, 2012

實做安裝FreeBSD 9.0R root on ZFS

基本上本文幾乎和參考資料相近, 差異在部份分割採用 copies=2. 由於實體機器上並沒有光碟機, 所以使用 USB 外接光碟機啟動 LiveCD.
0. 啟動安裝環境
0.1.  Boot 選單直接選 1. Boot
0.2. 中途若遇到找不到光碟機, 請下 cd9660:/dev/cd0 ro .
0.3. 出現安裝選項時, 請選 LiveCD.

1. 硬體
1.1. 系統硬碟 16GB SLC SSD. 快取硬碟 MLC 80GB SSD. 資料硬碟 1TB * 3.
1.2. 預計在系統上啟用 copies=2.

2. 建立 GPT 分割及 root on ZFS
gpart show ada0
gpart destroy ada0
gpart create -s GPT ada0
gpart add -b 34 -s 128 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gnop create -S 4096 /dev/gpt/disk0
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot
3. 建立子目錄
zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var

zfs create -o compression=on   -o exec=on  -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash

zfs create -o exec=off -o setuid=off -o copies=2 zroot/var/db
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o exec=off -o setuid=off zroot/var/run

zfs create -o compression=lzjb -o exec=on  -o setuid=off -o copies=2 zroot/var/db/pkg
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off -o copies=2 zroot/var/mail
zfs create -o compression=lzjb -o exec=on  -o setuid=off zroot/var/tmp
4. 建立 swap
zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap
5. 處理暫存目錄及home目錄
chmod 1777 /mnt/tmp
cd /mnt; ln -s usr/home home
chmod 1777 /mnt/var/tmp
6. 複製系統檔案
sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do(cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache
7. 處理開機設定
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
touch /mnt/etc/fstab
8. /tmp 改用 TMPFS
8.1. 重新開機後進 single user mode
mount -u /
swapon -a
zfs mount -a
zfs rename zroot/tmp zroot/tmpold
mkdir -p /tmp
chmod 1777 /tmp
8.2. 在 /etc/fstab 加上
# Device        Mountpoint      FStype  Options         Dump    Pass#
tmpfs           /tmp            tmpfs   rw              0       0

參考資料 :
1. ROOT ON ZFS FREEBSD 9 (NON LEGACY MOUNTPOINT – 4K OPTIMIZED)

Thursday, July 26, 2012

ZFS zpool 匯出(export)及匯入(import)

為啥會出現這個問題, 並不是因為換機器. 而是之前 BIOS 都是使用 SATA mode, 導致有兩個 port 的硬碟跑 Ultra DMA 133. 因為不熟 ZFS, 不敢亂玩(其實是怕資料沒了).
今天就大膽一點直接上了, 重開機後發現 BIOS 只抓到四個硬碟, 心裡涼了一半. 不過在 Unix* 環境似乎還可以期待一下系統自己辨識硬碟. 進了 single user mode 看了一下 dmesg 及 /dev, 發現硬碟全數找到了. 只是硬碟代號有部份異動, 剛好動到的都是 ZFS 的硬碟.
立馬回頭改回原設定, 一樣也是進 single user mode 下
zpool export POOL_NAME

再做一次轉換為 AHCI, 換好了以後再進 single user mode. 此時用 zpool import 去偵測發現只找到一個 STO_NAME. 一時原因無法理解, 本來想要就放棄了. 仔細比較磁碟代號, 發現有重複的磁碟代號出現(adX之前是 POOL1 現在變成是 POOL2). 趕緊先執行
zpool import POOL1

重新開機後再用 zpool import 掃描, 果真出現 POOL2. 再執行
zpool import POOL2

收工.

參考資料 :
1. Migrating ZFS Storage Pools Chapter 4. Managing ZFS Storage Pools

ZFS儲存設備更換硬碟

只能說硬碟不能放在太熱的環境(硬碟工作溫度攝氏 45 ~ 58 度), 否則企業級硬碟還是一樣會出問題.這個硬碟倒不是完全讀不到, 只是三不五時就會出現 smartd 的警告訊息.

SMART error (CurrentPendingSector) detected on host: XXXXXX.daemonland.org

This email was generated by the smartd daemon running on:

   host name: XXXXXX.daemonland.org
  DNS domain: daemonland.org
  NIS domain: 

The following warning/error was logged by the smartd daemon:

Device: /dev/ad10, 3 Currently unreadable (pending) sectors


For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.

只好將備用的 HITACHI 2TB 硬碟上線, 換 WD 硬碟下來維修. 順便打開系統的 AHCI.


1.更換 ZFS RAIDZ1 硬碟
因為沒熱插拔可用所以就重新開機進 single user mode. 然後依照步驟進行:
# zpool offline POOL_NAME DISKID
關機後, 更換故障硬碟.
# zpool online POOL_NAME DISKID
# zpool replace POOL_NAME DISKID

經過漫長的 resilvering (14小時, 可以執行 zpool status POOL_NAME 檢查進度), 終於完成更換硬碟. 沒有機會測試在多使用者模式下, 系統效能會是什麼情景.

2. 設定 AHCI
很簡單只要在 /boot/loader.conf 加上
ataahci_load="YES"
很訝異重新開機後磁碟機代號竟然沒有改變(沒有從 adX 變成 adaX).

參考資料 :
1. ZFS: how to replace a dead disk?
2. [Solved] ZFS cannot replace failed drive

Monday, August 15, 2011

How to create ZFS storage in FreeBSD 7.4

1. Edit /etc/rc.conf. Add
zfs_enable="YES"

2. Shutdown machine and plug all hard drives.

3. Create storage pool
#   zpool create PoolName raidz ad1 ad2 ad3 ...

4. Create volume
# zfs create PoolName/VolumeName

5. Set mount point
# zfs set mountpoint=/Path/To/Mount/Point PoolName/VolumeName

6. Setup daily check
# echo 'daily_status_zfs_enable="YES"' >> /etc/periodic.conf

7. Setup volume compression
# zfs set compression=gzip PoolName/VolumeName

8. Check ZFS pool status
# zpool status

9. Check ZFS command history
# zpool history

10. Check ZFS mounting point
# zfs get mountpoint

11. Setup volume quota
# zfs set quota=QuotaSize PoolName/VolumeName

12. Show volume setting
# zfs get all PoolName/VolumeName

13. Show ZFS I/O status
# zfs iostat 1

Saturday, August 13, 2011

使用 crontab 管理 ZFS snapshot

近期終於將之前延遲已久的 ZFS storage 計畫實現. 已試過當 raidz 硬碟發生問題時, 使用 zfs replace 更換硬碟. 當然就會想用 snapshot 這個重要的功能. 所以就動手寫了簡單的 shell script 搭配 crontab 做自動化管理.

1. 簡單指令
觀察 zpool 清單, 狀態
   # zpool list
   # zpool status

   觀察 ZFS volume 清單
   # zfs list

   觀察 snapshot 清單 (/usr/ports/sysutils/freebsd-snapshot)
   # snapshot list

2. shell script
#!/usr/local/bin/bash

PATH=$PATH

LOGFILE="/var/log/snapshotfs.log"
TODAY=`date +"%Y%m%d"`
CHKPARA=0

if [ -n "$1" ]; then
 fstype="$1"
 shift
else
 CHKPARA=1
fi


if [ -n "$1" ]; then
 sstype="$1"
 shift
else
 CHKPARA=1
fi


if [ -n "$1" ]; then
 maxnum="$1"
 shift
else
 CHKPARA=1
fi


if [ -n "$1" ]; then
 volpath="$1"
 shift
else
 CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
 echo "Usage: $0 fstype sstype maxnum /filesystem|volume_path"
 echo "Example: $0 zfs [H|D|W|M|Q|Y] 7 /databackup/backup"
 exit 1
else
 touch $LOGFILE
 echo `date +"%Y/%m/%d %H:%M:%S"` >> $LOGFILE
 echo "$0 $fstype $sstype $maxnum $volpath" >> $LOGFILE
 echo "Check parameters for taking a snapshot of $volpath($fstype)." >> $LOGFILE
fi

if [ -d $volpath ]; then
 if [ $fstype = 'zfs' ]; then
  if [ -d $volpath/.zfs/snapshot ]; then
   echo "Check path : OK" >> $LOGFILE
  else
   echo "Check path : Failed, snapshot folder does not exist." >> $LOGFILE
   exit 1
  fi
 elif [ $fstype = 'ufs' ]; then
  if [ -d $volpath/.snap ]; then
   echo "Check path : OK" >> $LOGFILE
  else
   echo "Check path : Failed, snapshot folder does not exist." >> $LOGFILE
   exit 1
  fi
 else
  echo "Unsupported file system $fstype" >> $LOGFILE
  exit 1
 fi
else
 echo "Check path : Failed, filesystem|volume_path does not exist." >> $LOGFILE
 exit 1
fi

if [ $maxnum -gt 0 ]; then
 echo "Check maxnum : OK" >> $LOGFILE
else
 echo "Check maxnum : Failed, must input positive integer." >> $LOGFILE
        exit 1
fi

if [ $fstype = 'zfs' ]; then
 echo "ZFS" >> $LOGFILE

 case $sstype in
 H)  echo "Hourly Snapshot" >> $LOGFILE
   TODAY=`date +"%Y%m%d%H"`
   ;;
 D)  echo "Daily Snapshot" >> $LOGFILE
   ;;
 W)  echo "Weekly Snapshot" >> $LOGFILE
   ;;
 M)  echo "Monthly Snapshot" >> $LOGFILE
   ;;
 Q)  echo "Quaterly Snapshot" >> $LOGFILE
   ;;
 Y)  echo "Yearly Snapshot" >> $LOGFILE
   ;;
 *)  echo "sstype must be H | D | W | M | Q | Y " >> $LOGFILE
   exit 1
 esac

 #Snapshot
 if [ -d $volpath/.zfs/snapshot/$sstype$TODAY ]; then
  echo "Had snapshot($volpath $sstype$TODAY)." >> $LOGFILE
 else
  echo "Take a snapshot of $volpath." >> $LOGFILE
  zfs snapshot `zfs list -H -o name ${volpath}`@${sstype}${TODAY}
 fi

 #Remove more old snapshot

 KEEPSSNUM=$maxnum
 REALSSNUM=`ls $volpath/.zfs/snapshot/ | grep $sstype | wc -l`
 RMSSNUM=`expr $REALSSNUM - $KEEPSSNUM`
 RMSSID=""

 if [ $RMSSNUM -gt 0 ]; then
  for RMSSID in `ls $volpath/.zfs/snapshot/ | grep $sstype | sort -n | head -n $RMSSNUM | sed -e 's/\///'`
  do
   if [ -d $volpath/.zfs/snapshot/$RMSSID ]; then
    echo "Remove old snapshot($volpath $RMSSID)." >> $LOGFILE
    zfs destroy `zfs list -H -o name ${volpath}`@$RMSSID
   else
    echo "Old snapshot($volpath $RMSSID) does not exist." >> $LOGFILE
   fi
  done
 fi

elif [ $fstype = 'ufs' ]; then
 echo "UFS" >> $LOGFILE

 case $sstype in
 H) echo "Hourly Snapshot" >> $LOGFILE
  sskey='Hourly'
  ;;
 D) echo "Daily Snapshot" >> $LOGFILE
  sskey='Daily'
  ;;
 W) echo "Weekly Snapshot" >> $LOGFILE
  sskey='Weekly'
  ;;
 M) echo "Monthly Snapshot" >> $LOGFILE
  sskey='Monthly'
  ;;
 Q) echo "Quaterly Snapshot" >> $LOGFILE
  sskey='Quaterly'
  ;;
 Y) echo "Yearly Snapshot" >> $LOGFILE
  sskey='Yearly'
  ;;
 *) echo "sstype must be H | D | W | M | Q | Y " >> $LOGFILE
  exit 1
 esac
 
 #Snapshot
 echo "Take a snapshot UFS ${volpath}" >> $LOGFILE
 /usr/local/sbin/snapshot make -g${maxnum} ${volpath}:${sskey}
else
 echo "Unsupported file system $fstype" >> $LOGFILE
fi
3. crontab 排程自動化

### ZFS snapshot
### Hourly
45      *       *       *       *       root    /PATH/TO/zfs_snapshot_hourly.sh
### 3 Hourly
50      */3     *       *       *       root    /PATH/TO/zfs_snapshot_3hourly.sh
### Daily
0       23      *       *       *       root    /PATH/TO/zfs_snapshot_daily.sh
### Weekly
20      23      *       *       1       root    /PATH/TO/zfs_snapshot_weekly.sh
### Monthly
40      23      1       *       *       root    /PATH/TO/zfs_snapshot_monthly.sh
### Quaterly
53      23      1       1,4,7,10        *       *       root    /PATH/TO/zfs_snapshot_quaterly.sh
### Yearly
55      23      31      12      *       root    /PATH/TO/zfs_snapshot_yearly.sh


#!/usr/local/bin/bash

PATH=$PATH

### ZFS snapshot               SSTYPE SSCOUNT
### H for hourly or 3hourly ex: H 8
### D for daily             ex: D 7
### W for weekly            ex: W 4
### M for monthly           ex: M 6
### Q for quaterly          ex: Q 4
### Y for yearly            ex: Y 3

/root/root_scripts/snapshotfs.sh zfs SSTYPE SSCOUNT /zfs-volumn/path1
/root/root_scripts/snapshotfs.sh zfs SSTYPE SSCOUNT /zfs-volumn/path2
/root/root_scripts/snapshotfs.sh zfs SSTYPE SSCOUNT /zfs-volumn/path3