Jengyic's 正義部落: FreeBSD

Showing posts with label FreeBSD. Show all posts

Monday, December 04, 2017

IBM X200s 改裝 Intel 5300 無線網卡

無線網路卡買的是 Intel Wifi link 5300

由於有鎖網卡, 請選擇支援自己筆電型號的網卡

直接將原無線網卡取代. 使用旁邊的卡槽無法正常找到網路卡.

在 /boot/loader.conf 加上下列設定在開機時載入核心模組

if_iwn_load="YES"

在 /etc/rc.conf 加入下列設定啟動無線網卡

wlans_iwn0="wlan0"

ifconfig_wlan0="WPA DHCP"

Sunday, December 03, 2017

FreeBSD 11.1 local_unbound 設定

為何想要用 local_unbound? 主要是為了 DNSSEC 檢驗能從本機開始檢查,
既然各軟體要外掛實在是有難度. 所以就安裝本機快取並使用已經啟動
DNSSEC 檢查的 cache DNS.

以下就是簡單的啟動步驟:
設定檔 /etc/rc.conf
local_unbound_enable="YES"

local_unbound 設定檔目錄 /var/unbound

root@bsd11x200s:/var/unbound # ls -la

total 36

drwxr-xr-x 3 unbound unbound 8 Dec 3 23:08 .

drwxr-xr-x 26 root wheel 26 Dec 3 21:27 ..

drwxr-xr-x 2 unbound unbound 3 Dec 3 23:07 conf.d

-rw-r--r-- 1 root unbound 193 Oct 11 09:54 control.conf

-rw-r--r-- 1 root unbound 104 Dec 3 22:56 forward.conf

-rw-r--r-- 1 root unbound 189 Oct 11 09:54 lan-zones.conf

-rw-r--r-- 1 unbound unbound 1252 Dec 3 23:08 root.key

-rw-r--r-- 1 root unbound 389 Oct 11 09:54 unbound.conf

設定轉送查詢的 cache DNS

root@bsd11x200s:/var/unbound # cat forward.conf

# Generated by resolvconf

forward-zone:

name: "."

forward-addr: 8.8.8.8

forward-addr: 192.168.xxx.xxx

設定一些 unbound 參數

root@bsd11x200s:/var/unbound # cat conf.d/options.conf

server:

num-threads: 3

interface: 0.0.0.0

access-control: 127.0.0.0/8 allow

harden-glue: yes

harden-dnssec-stripped: yes

harden-below-nxdomain: yes

harden-referral-path: yes

啟動服務

root@bsd11x200s:/var/unbound # service local_unbound start

root@bsd11x200s:/var/unbound # cat /etc/resolv.conf

nameserver 127.0.0.1

options edns0

測試

(1) host

root@bsd11x200s:/var/unbound # host www.google.com 127.0.0.1

Using domain server:

Name: 127.0.0.1

Address: 127.0.0.1#53

Aliases:

www.google.com has address 216.58.200.36

www.google.com has IPv6 address 2404:6800:4008:801::2004

(2) nslookup

root@bsd11x200s:/var/unbound # nslookup www.google.com 127.0.0.1

Server: 127.0.0.1

Address: 127.0.0.1#53

Non-authoritative answer:

Name: www.google.com

Address: 172.217.160.100

Name: www.google.com

Address: 2404:6800:4008:801::2004

(3) Dig 測試 DNSSEC

root@bsd11x200s:/var/unbound # dig +dnssec @127.0.0.1 www.icann.org A

; <<>> DiG 9.11.2 <<>> +dnssec @127.0.0.1 www.icann.org A

; (1 server found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65414

;; flags: qr rd ra ad; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags: do; udp: 4096

;; QUESTION SECTION:

;www.icann.org. IN A

;; ANSWER SECTION:

www.icann.org. 3566 IN CNAME www.vip.icann.org.

www.icann.org. 3566 IN RRSIG CNAME 7 3 3600 20171214150526 20171123111933 7500 icann.org. i5KCfOv/+c/xIaRxmQrqTItEtlGkTe9RWoh2gKh5Rh/a431n2CLLNiZ2 8xpuDXoqb2KQlAz4JBhyydFb2ilPCJIPXFbiQbUksuDV6zHdYu+PzlUC hbxRkBhRD2CIPOc227igVbp3io4oTqlzaja4P0gq268yoVEOPb2/2sSs DzFDV80=

www.vip.icann.org. 29 IN A 192.0.32.7

www.vip.icann.org. 29 IN RRSIG A 7 4 30 20171208095813 20171201095813 6887 vip.icann.org. k4FaKFgAcblg0yRPMyP1TvNcXmgI9d1GXM4YQ9jelGeCro2I+OQW+BFg OKQgmjcnh3M7NiZBW2vQQF000iKWS938B/YeiODQuun+kv9ANaiCEipD /0FFkjecpgwuMCCy2ommmRsWSlv4484Sdw7KRW8S28SZj8jonCqzIDct rVQ=

www.vip.icann.org. 29 IN RRSIG A 7 4 30 20171208095813 20171201095813 4532 vip.icann.org. FoSTut3HaWz1SEHtxReWh7IuOHOv4kCAEIuz1zPQPQVvNDU6WHenVF4h YSJOSLgyLwZ9P1MsH8PGPJEpCJr6jOR1Gl7QF24UezSj5PXnoinPLyjG Zm8/3DcM6UcrZhwmm9wRTSOuGv+LAmiU9Ab0WUfKEKR/vMnj4qn5HR+Y QOI=

;; Query time: 315 msec

;; SERVER: 127.0.0.1#53(127.0.0.1)

;; WHEN: Sun Dec 03 23:24:36 CST 2017

;; MSG SIZE rcvd: 598

至於想要了解什麼是 DNSSEC 請先看

DNSSEC – What Is It and Why Is It Important?

https://www.icann.org/resources/pages/dnssec-qaa-2014-01-29-en

以下是 TLD 支援 DNSSEC 的現況

TLD DNSSEC Report

http://stats.research.icann.org/dns/tld_report/

參考資料

1. How To Set Up the Unbound Caching DNS Resolver on FreeBSD 10.1 or 10.2

https://www.digitalocean.com/community/tutorials/how-to-set-up-the-unbound-caching-dns-resolver-on-freebsd-10-1

2. unbound.conf - Unbound configuration file.
https://www.unbound.net/documentation/unbound.conf.html

Sunday, December 30, 2012

使用 shell script 透過 SSH 在遠端伺服器執行程式

這隻程式只是個簡單的實做連線前偵測遠端伺服器是否可以 ping 到及服務埠是否有開啟. 程式碼如下

FreeBSD下用 shell script 建立ZFS snapshot 更新版

此次更新的目的是讓 ZFS 不使用 ls /PATH/.zfs/snapshot 目錄判斷快照數量(為了避開 Bad file descriptor 問題). 中間版本曾經使用 /usr/local/bin/snapshot list 進行, 但是速度較慢. 好在這個檔案是 shell script. 看了一下 op_list 函式的內容看到取得目前已有的快照方法, 也了解到這隻程式慢在還要查詢及計算使用空間等數據. 所以將查詢改用 zfs list -H -t snapshot -o name | grep "ZPOOL_NAME/ZVOL_NAME@". (ZPOOL_NAME/ZVOL_NAME 可以使用 zfs list -H -o name ${volpath} 查詢得到)

使用 SSHFS 掛載遠端目錄

1. 安裝 SSHFS
FreeBSD 下請安裝 /usr/ports/sysutils/fusefs-sshfs
Ubuntu/Debian 下請 apt-get install sshfs

Mount :
sshfs RemoteServerAccountName@SSHServerName:/Remote/path /Mount/path -o follow_symlinks

Umount :
FreeBSD : umount /Mount/path
Ubuntu/Debian : fusermount -u /Mount/path

2. 使用掛載 shell script
目前暫時不做作業系統版本判斷. 所以有以下的兩組 shell script:

讓 cron 可以每隔幾秒執行 shell script

之前使用每五分鐘檢查是否有嘗試入侵的機器, 發現有些機器會一下送來一堆連線. 所以只好將執行時間降到以秒為單位. 由於檢查用的 shell script 約在幾秒內完成, 所以就用下列的 shell script 來達成.

在 FreeBSD 建立樣版功能的 jail 環境

0. 準備步驟 :
0.0. 準備 Host 的網路環境
0.0.0. 綁 IP 到 Host : 編輯 /etc/rc.conf 加上 IP alias 設定
ifconfig_em0_alias1="192.168.66.66 netmask 255.255.255.255"

0.0.1. 設定 natd (如果是只有一片網卡的機器就可以不用, 此步驟適用機器有跨在兩個網路(一內一外)上)
Edit /etc/rc.conf

gateway_enable="YES"

firewall_enable="YES"
firewall_type="/etc/my.firewall"
firewall_script="/etc/my.firewall"
firewall_logging="YES"

natd_enable="YES"
natd_interface="re0"
natd_flags="-l -f /etc/natd.conf"

Edit /etc/my.firewall (請依照實際需求更改)

#!/bin/sh -

fwcmd="/sbin/ipfw"

INTIF="re0"

LANIF0="em0"

${fwcmd} -f flush
#For NATD
${fwcmd} add 00030 divert natd all from any to any via ${INTIF}
...
${fwcmd} add 65000 pass all from any to any

Edit /etc/natd.conf

use_sockets yes
same_ports yes
dynamic yes

Edit /etc/services

natd 8668/divert # Network Address Translation

接著 Host 重新啟動.

0.1. 在 /usr/src 編譯整個環境.
# cd /usr/src; make buildworld

1. 設定樣板存放位置且安裝樣板

# zfs create storage/jail
# zfs set mountpoint=/storage/jail storage/jail
# zfs set dedup=on storage/jail
# zfs set quota=300G storage/jail

# zfs create storage/jails
# zfs set mountpoint=/storage/jails storage/jails
# zfs set dedup=on storage/jails
# zfs set quota=300G storage/jails

# mkdir -p /storage/jail/mroot

# cd /usr/src
# make installworld DESTDIR=/storage/jail/mroot

2. 複製 src 及 ports
# cd /storage/jail/mroot
# mkdir usr/ports
# portsnap -p /storage/jail/mroot/usr/ports fetch extract
# cpdup /usr/src /storage/jail/mroot/usr/src

3. 建立樣板環境
# mkdir /storage/jail/skel /storage/jail/skel/home /storage/jail/skel/usr-X11R6 /storage/jail/skel/distfiles /storage/jail/skel/packages
# mv etc /storage/jail/skel
# mv usr/local /storage/jail/skel/usr-local
# mv tmp /storage/jail/skel
# mv var /storage/jail/skel
# mv root /storage/jail/skel

4. 產生設定檔
# mergemaster -t /storage/jail/skel/var/tmp/temproot -D /storage/jail/skel -i
# cd /storage/jail/skel
# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev

5. 建立個別環境可寫入的目錄
#cd /storage/jail/mroot
#mkdir s
#ln -s s/etc etc
#ln -s s/home home
#ln -s s/root root
#ln -s ../s/usr-local usr/local
#ln -s ../s/usr-X11R6 usr/X11R6
#ln -s ../../s/distfiles usr/ports/distfiles
#ln -s ../../s/packages usr/ports/packages
#ln -s s/tmp tmp
#ln -s s/var var

6. 建立 jailed O.S. 的 make.conf
# Add /storage/jail/skel/etc/make.conf
# WRKDIRPREFIX?= /s/portbuild

以下是每個 jailed O.S. 都要做一次

7. 設定 Host 每次開機為 jailed O.S. 掛上目錄
[FreeBSD 9 labdata2] (Why: 開機時因 ZFS 來不及掛載而無法在 /etc/fstab 處理)
Edit /etc/rc.local
### For jail
/sbin/mount -t nullfs -r /storage/jail/mroot /storage/jail/labdata2
/sbin/mount -t nullfs -rw /storage/jails/labdata2 /storage/jail/labdata2/s

[FreeBSD 8 localdata2]
Edit /etc/fstab

#/storage/jail/mroot /storage/jail/localdata2 nullfs ro 0 0
#/storage/jails/localdata2 /storage/jail/localdata2/s nullfs rw 0 0

8. 編輯 Host 的 /etc/rc.conf
###20121004
jail_enable="YES"
jail_set_hostname_allow="NO"
### 以下是清單, 有幾台 jailed O.S. 就登記幾台
jail_list="localdata2"
### 以下是每一台 jailed O.S. 的設定記得每一台都要有一組設定.(前四行是必要的)
jail_localdata2_hostname="localdata2.DOMAIN_NAME"
jail_localdata2_ip="192.168.66.66"
jail_localdata2_interface="em0"
jail_localdata2_rootdir="/storage/jail/localdata2"
jail_localdata2_devfs_enable="YES"
### 20121113 解決 netstat -rn 查詢問題
#jail_localdata2_devfs_ruleset="devfsrules_jail"
jail_localdata2_devfs_ruleset="devfsrules_jail2"
jail_localdata2_exec_start="/bin/sh /etc/rc"
jail_localdata2_exec_stop="/bin/sh /etc/rc.shutdown"
jail_localdata2_fdescfs_enable="YES"
jail_localdata2_procfs_enable="YES"

9. 建立系統目錄
mkdir /storage/jail/localdata2
mkdir /storage/jails/localdata2
cpdup /storage/jail/skel /storage/jails/localdata2
mkdir /storage/jail/localdata2/s

10. [Optional] 在 Host 編輯 /etc/sysctl.conf 開放 jailed O.S. 使用 ping.
Edit /etc/sysctl.conf
###20121113 For jail to fix ping icmp socket: Operation not permitted
security.jail.allow_raw_sockets=1

11. [Optional] 在 Host 編輯 /etc/default/devfs.rules 開放 jailed O.S. 使用 pts pty mem bpf
11.1. Edit /etc/default/devfs.rules
[devfsrules_unhide_login=3]
...
add path pts unhide
add path 'pts/*' unhide
add path pty unhide
add path 'pty/*' unhide

11.2. Edit /etc/devfs.rules
###
### http://forums.freebsd.org/archive/index.php/t-24581.html
### http://forums.freebsd.org/archive/index.php/t-5693.html
###
###

[devfsrules_unhide_mem=5]
add path mem unhide
add path kmem unhide

[devfsrules_unhide_bpf=6]
add path bpf unhide

# Devices usually found in a jail.
#
[devfsrules_jail2=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_mem
add include $devfsrules_unhide_bpf

12. 掛載目錄
[FreeBSD 9]

/sbin/mount -t nullfs -r /storage/jail/mroot /storage/jail/labdata2
/sbin/mount -t nullfs -rw /storage/jails/labdata2 /storage/jail/labdata2/s

[FreeBSD 8]
mount /storage/jail/localdata2
mount /storage/jail/localdata2/s

13. 啟動 jail
# /etc/rc.d/jail start
查詢啟動的 jailed O.S.
# jls
進入 jailed O.S.
# jexec #JID tcsh (或 jexec #JID /ur/bin/su -)

14. 編輯 jailed O.S. 的系統設定
Edit /etc/rc.conf
Edit /etc/resolv.conf
...

參考資料
1. FreeBSD 使用手冊 15.6 Jail 的應用
2. Forums.freebsd.org : [Solved] jails with vnet in rc.conf
3. Forums.freebsd.org : jail nat route

Saturday, September 15, 2012

自家用 ipfw log 統計

雖然有 security/ipfwcount 好用, 但是總想可以自由的分析 ipfw.log. 所以就寫了以下的 stupid script. 哈哈 ... 但是它可以用.

#!/usr/local/bin/bash
##
##  FreeBSD ipfw log analyzer
##

PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin:$PATH"

TS=`date +"%s"`
TMPFILE="/tmp/ana_ipfw-$TS"
TMPFILEsort="/tmp/ana_ipfw_sort-$TS"
LC_ALL=C
LANG=C

### Default options
datelimit=no
portlimit=no
iplimit=no
verbose=no
sorted=no
onlydata=no
datestr=""
logfile="/var/log/ipfw.log"

### Help
usage () {
 echo "Usage: $0 [-p] [-i] [-v] [-s] [-o] [-d date_limit] [filename]"
 echo "   -p  port analysis"
 echo "   -i  IP address analysis"
 echo "   -d  date format"
 echo "   +DDAY   from DDAY to today"
 echo "   +-DDAY  DDAY days ago"
 echo "   YYYYMMDD-YYYYMMDD  from first YYYYMMDD to last YYYYMMDD"
 echo "   total   all data records in the log file"
 echo "   -s  sorted by count"
 echo "   -o  only data and disable verbose"
 echo "   -v  enable verbose messages"
 echo "   filename default : /var/log/ipfw.log"
}
if [ $# -eq 0 ]; then
 usage
 exit 1
fi

### Get arguments
args=`getopt dpivso $*`

if [ $? != 0 ]; then
 echo "$0:ERROR"
 exit 2
fi

set -- $args

for arg
do
 case "$arg" in
  -d )  datelimit=yes; shift ;;
  -p )  portlimit=yes; shift ;;
  -i )  iplimit=yes; shift ;;
  -v )  verbose=yes; shift ;;
  -s )  sorted=yes; shift ;;
  -o )  onlydata=yes; shift ;;
  -- )  datestr=$2; shift; shift ;;
 esac
done

if [ $datelimit == "no" ]; then
 if [ ! -z $datestr ]&&[ -f $datestr ]; then
  logfile=$datestr
 fi
else
 if [ ! -z $datestr ]&&[ ! -z $1 ]; then
  logfile=$1
 fi
fi

if [ ! -f $logfile ]; then
 echo "Error : Log file $logfile does not existed."
 exit 1
fi

if [ $onlydata == "yes" ]; then
 verbose=no
fi

### For debug
if [ $verbose == "yes" ]; then
 echo "Parameters"
 echo $args
 echo "Date $datelimit($datestr) Port $portlimit IP $iplimit Verbose $verbose Sort $sorted Logfile $logfile"
 echo ""
fi

### Generate date limitation
if [ $datelimit == "no" ]; then
 dtmp="0"
 datearray=($dtmp)
else
 if [ ! -z `echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }'` ]; then
  if [ ! -z `echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }' | awk 'BEGIN { FS = "-" } ; { print $2 }'` ]; then
   datearray=(`echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }' | awk 'BEGIN { FS = "-" } ; { print $2 }'`)
  else
   i=`echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }'`
   dtmp=""
   for((; i >= 0; i--))
   do
    dtmp="$dtmp $i"
   done
   datearray=($dtmp)
  fi
 elif [ "$datestr" == "total" ]; then
  datestr="total"
 elif [ -z $datestr ]; then
  dtmp="0"
  datearray=($dtmp)
 else
  if [[ $datestr != *[!0-9\-]* ]]; then
   bdate=`echo $datestr | awk 'BEGIN { FS = "-" } ; { print $1 }'`
   edate=`echo $datestr | awk 'BEGIN { FS = "-" } ; { print $2 }'`
  else
   bdate=`date +"%Y%m%d"`
   edate=`date +"%Y%m%d"`
  fi

  todayts=`date -j "+%s"`
  bdatets=`date -j -f "%Y%m%d" "$bdate" "+%s"`
  edatets=`date -j -f "%Y%m%d" "$edate" "+%s"`

  if [ $bdatets -gt $todayts ]; then
   echo "Error : $bdate is future. I can not analyze it."
   exit 1
  else
   ib=`expr \( $todayts - $bdatets \) / 86400`
  fi
  if [ $edatets -gt $todayts ]; then
   if [ $verbose == "yes" ]; then
    echo "Warning : $edate is future. But I can replace it with today("`date +"%Y%m%d"`")."
    echo ""
   fi
   ie=0
  else
   ie=`expr \( $todayts - $edatets \) / 86400`
  fi

  dtmp=""
  for((i = $ib; i >= $ie; i--))
  do
   dtmp="$dtmp $i"
  done
  datearray=($dtmp)
 fi
fi

### Create tmp file
if [ $verbose == "yes" ]; then
 echo "Create $TMPFILE"
 echo ""
fi
touch $TMPFILE

### Get log
if [ $verbose == "yes" ]; then
 echo "Get log from $logfile"
fi
if [ $datelimit == "no" ]; then
 if [ $verbose == "yes" ]; then
  date +"%b %e"
 fi
 keyfilter=`date +"%b %e"`
 grep "$keyfilter" $logfile >> $TMPFILE
elif [ "$datestr" == "total" ]; then
 cat $logfile > $TMPFILE
else
 for dnum in ${datearray[@]}
 do
  if [ $verbose == "yes" ]; then
   date -v-${dnum}d +"%b %e"
  fi
  keyfilter=`date -v-${dnum}d +"%b %e"`
  grep "$keyfilter" $logfile >> $TMPFILE
 done
fi
if [ $verbose == "yes" ]; then
 echo ""
fi

if [ $verbose == "yes" ]; then
 echo "Count lines in $TMPFILE"
 wc -l $TMPFILE
 echo ""
fi

### Port Analysis
if [ $portlimit == "yes" ]; then
 if [ $sorted == "yes" ]; then
  rm -f $TMPFILEsort
  touch $TMPFILEsort
 fi

 if [ $onlydata == "no" ]; then
  echo "[[[ Port Analysis ]]]"
  printf "%12s\t%12s\n" "Port Number" "Count"
  printf "%12s\t%12s\n" "------------" "--------"
 fi
 for portnum in `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { if( $11=="\-\>" ){ split($12, p, ","); print p[1] } else { print $11 } }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $2 }' | sort -u -n`
 do
  if [ $sorted == "yes" ]; then
   printf "%12s\t%12s\n" "$portnum" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { if( $11=="\-\>" ){ split($12, p, ","); print p[1] } else { print $11 } }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $2 }' | grep $portnum | wc -l` >> $TMPFILEsort
  else
   printf "%12s\t%12s\n" "$portnum" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { if( $11=="\-\>" ){ split($12, p, ","); print p[1] } else { print $11 } }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $2 }' | grep $portnum | wc -l`
  fi
 done
 if [ $sorted == "yes" ]; then
  cat $TMPFILEsort | awk 'BEGIN { FS = " " } ; { printf ( "%12s\t%12s\n", $1, $2) }' | sort -rgk2 > /tmp/ttt-$TS
  cat /tmp/ttt-$TS > $TMPFILEsort
  rm -f /tmp/ttt-$TS
  cat $TMPFILEsort
 fi
fi

### Client IP Analysis
if [ $iplimit == "yes" ]; then
 if [ $sorted == "yes" ]; then
  rm -f $TMPFILEsort
  touch $TMPFILEsort
 fi

 if [ $onlydata == "no" ]; then
  echo "[[[ Client IP Analysis ]]]"
  printf "%16s\t%12s\n" "IP Address" "Count"
  printf "%16s\t%12s\n" "----------------" "--------"
 fi
 for ipaddr in `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { print $10 }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $1 }' | sort -u -n`
 do
  if [ $sorted == "yes" ]; then
   printf "%16s\t%12s\n" "$ipaddr" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { print $10 }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $1 }' | grep $ipaddr | wc -l` >> $TMPFILEsort
  else
   printf "%16s\t%12s\n" "$ipaddr" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { print $10 }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $1 }' | grep $ipaddr | wc -l`
  fi
 done
 if [ $sorted == "yes" ]; then
  cat $TMPFILEsort | awk 'BEGIN { FS = " " } ; { printf ( "%16s\t%12s\n", $1, $2) }' | sort -rgk2 > /tmp/ttt-$TS
  cat /tmp/ttt-$TS > $TMPFILEsort
  rm -f /tmp/ttt-$TS
  cat $TMPFILEsort
 fi
fi

### Remove tmp file
if [ $verbose == "yes" ]; then
 echo ""
 echo "Remove $TMPFILE"
fi
rm -f $TMPFILE
if [ $sorted == "yes" ]; then
 if [ $verbose == "yes" ]; then
  echo "Remove $TMPFILEsort"
 fi
 rm -f $TMPFILEsort
fi

使用方法如下 :

Usage: /Path/to/ana_ipfw.sh [-p] [-i] [-v] [-s] [-o] [-d date_limit]  [filename]
   -p           port analysis
   -i           IP address analysis
   -d           date format
                        +DDAY   from DDAY to today
                        +-DDAY  DDAY days ago
                        YYYYMMDD-YYYYMMDD  from first YYYYMMDD to last YYYYMMDD
                        total   all data records in the log file
   -s           sorted by count
   -o           only data and disable verbose
   -v           enable verbose messages
   filename     default : /var/log/ipfw.log

分析範例如下 :
今天的 port 分析

# ./ana_ipfw.sh -p
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
          21               4
          22              14
          25              13
          80              30
         111               2
         199               2
         389               2
         443               8
         873               2
        2049               2
        3000              36
        3306              11
        5800               2
        5900               5
        6000               2
        6001               2

今天的 port 分析(按照數量排序)

# ./ana_ipfw.sh -ps
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
        3000              36
          80              30
          22              14
          25              13
        3306              12
         443               8
        5900               5
          21               4
        6001               2
        6000               2
        5800               2
        2049               2
         873               2
         389               2
         199               2
         111               2

最近三天的 port 分析

# ./ana_ipfw.sh -p -d +3
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
          21               7
          22              20
          25              41
          80              72
         111               2
         199               2
         389               2
         443              14
         873               2
        2049               2
        3000              72
        3306              57
        5800               2
        5900              12
        5901               3
        6000               7
        6001               2

最近三天的 port 分析(按照數量排序)

# ./ana_ipfw.sh -p -d +3 -s
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
        3000              72
          80              72
        3306              57
          25              41
          22              20
         443              14
        5900              12
        6000               7
          21               7
        5901               3
        6001               2
        5800               2
        2049               2
         873               2
         389               2
         199               2
         111               2

最後開進工程模式(用來修 script 用, 應該還有 bug.).

# ./ana_ipfw.sh -pv
Parameters
-p -v --
Date no() Port yes IP no Verbose yes Sort no Logfile /var/log/ipfw.log

Create /tmp/ana_ipfw-1347716436

Get log from /var/log/ipfw.log
Sep 15

Count lines in /tmp/ana_ipfw-1347716436
     495 /tmp/ana_ipfw-1347716436

[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
          21               4
          22              14
          25              13
          80              30
         111               2
         199               2
         389               2
         443               8
         873               2
        2049               2
        3000              36
        3306              12
        5800               2
        5900               5
        6000               2
        6001               2

Remove /tmp/ana_ipfw-1347716436

Thursday, August 30, 2012

FreeBSD 下 NTFS 效能

由於建立了儲存設備二號機, 之前放在 Windows XP 下磁碟陣列的資料也備份到 USB 外接硬碟. 就想將備份資料放到二號機(FreeBSD 9.0R + ZFS). 心想系統已經有內建的 mount_ntfs 就直接拿來用.
mount_ntfs -C Big-5 /dev/da0s1 /mnt
沒想到 rsync 下去, 速度竟然只有 1MB/s.
檢查確認 da0 是掛在 usbus1(on ehci0) 下

...

usbus1: EHCI version 1.0

usbus1:  on ehci0

...

usbus1: 480Mbps High Speed USB v2.0

...

ugen1.1:  at usbus1

uhub1:  on usbus1

...

ugen1.2:  at usbus1

umass0:  on usbus1

umass0:  SCSI over Bulk-Only; quirks = 0x4000

umass0:13:0:-1: Attached to scbus13

da0 at umass-sim0 bus 0 scbus13 target 0 lun 0

da0:  Fixed Direct Access SCSI-2 device

da0: 40.000MB/s transfers

da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)

看過這篇討論覺得應該嘗試一下 sysutils/fusefs-ntfs.
沒想到效能差這麼多, 同樣使用 rsync 目前速度已經飆到平均 13MB/s , 最大值有看到 22MB/s. 總算可以快速將資料備份. 放在單一硬碟上還真是蠻擔心的.

Thursday, August 02, 2012

測試 ZFS copies 在單顆磁碟下的復原能力.

感謝 Kuli 學長提供良好的測試方法 - 使用 memory disk 測試. 測試指令如下 :

1. 準備階段
1.1. 建立 zpool test 及測試 vol 1 及 2

# mdconfig -a -t swap -s 66m
# zpool create test /dev/md0
# zfs create -o copies=1 test/1
# zfs create -o copies=2 test/2

1.2. 編輯測試檔案

# vi /test/1/test1.txt

輸入 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

# vi /test/2/test2.txt

輸入 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

1.3. 匯出 zpool test 並製作 memory disk dump file.

# zpool export test
# dd if=/dev/md0 of=/root/md0

2. 複製 dump file 並使用 hexedit 搜尋連續的 A 及 B 編輯檔案.

# cp -a /root/md0 /root/md0-edit
# hexedit md0

2.1. 搜尋有 36 個 A 的區塊將 A 改成 C. 進行 3. 觀察 ZFS 的反應.

2.2. 搜尋有 36 個 B 的區塊將 B 改成 D, 但只修改其中一組. 進行 3. 觀察 ZFS 的反應.

2.3. 搜尋有 36 個 B 的區塊, 其中一組將 B 改成 D, 另一組將 B 改成 E. 進行 3. 觀察 ZFS 的反應.

3. 將檔案寫回 memory disk

# dd if=/root/md0-edit of=/dev/md0 bs=1m
# zpool import -d /dev test
# zfs scrub test

4. 觀察結果:
4.1. 進行 2.1. 後, vol 1 出現無法修復的錯誤, 並拒絕使用該檔案.

4.2. 進行 2.2. 後, vol 2 發現錯誤並直接修正錯誤(不需 zfs scrub test), 且匯入後可正常使用該檔案. 此時將 zpool test 匯出後並製作 dump file. 可以觀察到手動變更的 D 已經被修改回 B.

4.3. 進行 2.3. 後, vol 2 出現無法修復的錯誤, 並拒絕使用該檔案.

5. 不過這是可修正的軟體錯誤, 如果發生硬體錯誤會是如何處理. 目前尚未想到好方法測試. 不過可偵測錯誤並顯示出來或許對系統管理就蠻有用的. 至少可以提早發現更換硬碟.

Saturday, July 28, 2012

實做安裝FreeBSD 9.0R root on ZFS

基本上本文幾乎和參考資料相近, 差異在部份分割採用 copies=2. 由於實體機器上並沒有光碟機, 所以使用 USB 外接光碟機啟動 LiveCD.
0. 啟動安裝環境
0.1. Boot 選單直接選 1. Boot
0.2. 中途若遇到找不到光碟機, 請下 cd9660:/dev/cd0 ro .
0.3. 出現安裝選項時, 請選 LiveCD.

1. 硬體
1.1. 系統硬碟 16GB SLC SSD. 快取硬碟 MLC 80GB SSD. 資料硬碟 1TB * 3.
1.2. 預計在系統上啟用 copies=2.

2. 建立 GPT 分割及 root on ZFS

gpart show ada0
gpart destroy ada0
gpart create -s GPT ada0
gpart add -b 34 -s 128 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

gnop create -S 4096 /dev/gpt/disk0
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

3. 建立子目錄

zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var

zfs create -o compression=on   -o exec=on  -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash

zfs create -o exec=off -o setuid=off -o copies=2 zroot/var/db
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o exec=off -o setuid=off zroot/var/run

zfs create -o compression=lzjb -o exec=on  -o setuid=off -o copies=2 zroot/var/db/pkg
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off -o copies=2 zroot/var/mail
zfs create -o compression=lzjb -o exec=on  -o setuid=off zroot/var/tmp

4. 建立 swap

zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap

chmod 1777 /mnt/tmp
cd /mnt; ln -s usr/home home
chmod 1777 /mnt/var/tmp

6. 複製系統檔案

sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do(cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

7. 處理開機設定

echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
touch /mnt/etc/fstab

8. /tmp 改用 TMPFS
8.1. 重新開機後進 single user mode

mount -u /
swapon -a
zfs mount -a
zfs rename zroot/tmp zroot/tmpold
mkdir -p /tmp
chmod 1777 /tmp

8.2. 在 /etc/fstab 加上

# Device        Mountpoint      FStype  Options         Dump    Pass#
tmpfs           /tmp            tmpfs   rw              0       0

參考資料 :
1. ROOT ON ZFS FREEBSD 9 (NON LEGACY MOUNTPOINT – 4K OPTIMIZED)

Thursday, July 26, 2012

ZFS zpool 匯出(export)及匯入(import)

為啥會出現這個問題, 並不是因為換機器. 而是之前 BIOS 都是使用 SATA mode, 導致有兩個 port 的硬碟跑 Ultra DMA 133. 因為不熟 ZFS, 不敢亂玩(其實是怕資料沒了).
今天就大膽一點直接上了, 重開機後發現 BIOS 只抓到四個硬碟, 心裡涼了一半. 不過在 Unix* 環境似乎還可以期待一下系統自己辨識硬碟. 進了 single user mode 看了一下 dmesg 及 /dev, 發現硬碟全數找到了. 只是硬碟代號有部份異動, 剛好動到的都是 ZFS 的硬碟.
立馬回頭改回原設定, 一樣也是進 single user mode 下

zpool export POOL_NAME

再做一次轉換為 AHCI, 換好了以後再進 single user mode. 此時用 zpool import 去偵測發現只找到一個 STO_NAME. 一時原因無法理解, 本來想要就放棄了. 仔細比較磁碟代號, 發現有重複的磁碟代號出現(adX之前是 POOL1 現在變成是 POOL2). 趕緊先執行

zpool import POOL1

重新開機後再用 zpool import 掃描, 果真出現 POOL2. 再執行

zpool import POOL2

收工.

參考資料 :
1. Migrating ZFS Storage Pools Chapter 4. Managing ZFS Storage Pools

ZFS儲存設備更換硬碟

只能說硬碟不能放在太熱的環境(硬碟工作溫度攝氏 45 ~ 58 度), 否則企業級硬碟還是一樣會出問題.這個硬碟倒不是完全讀不到, 只是三不五時就會出現 smartd 的警告訊息.

SMART error (CurrentPendingSector) detected on host: XXXXXX.daemonland.org

This email was generated by the smartd daemon running on:

   host name: XXXXXX.daemonland.org
  DNS domain: daemonland.org
  NIS domain: 

The following warning/error was logged by the smartd daemon:

Device: /dev/ad10, 3 Currently unreadable (pending) sectors


For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
No additional email messages about this problem will be sent.

只好將備用的 HITACHI 2TB 硬碟上線, 換 WD 硬碟下來維修. 順便打開系統的 AHCI.

1.更換 ZFS RAIDZ1 硬碟
因為沒熱插拔可用所以就重新開機進 single user mode. 然後依照步驟進行:

# zpool offline POOL_NAME DISKID

關機後, 更換故障硬碟.

# zpool online POOL_NAME DISKID
# zpool replace POOL_NAME DISKID

經過漫長的 resilvering (14小時, 可以執行 zpool status POOL_NAME 檢查進度), 終於完成更換硬碟. 沒有機會測試在多使用者模式下, 系統效能會是什麼情景.

2. 設定 AHCI
很簡單只要在 /boot/loader.conf 加上

ataahci_load="YES"

很訝異重新開機後磁碟機代號竟然沒有改變(沒有從 adX 變成 adaX).

參考資料 :
1. ZFS: how to replace a dead disk?
2. [Solved] ZFS cannot replace failed drive

Sunday, March 18, 2012

在 FreeBSD 8 對 UFS filesystem 做 snapshot

1. 在 crontab 定期做 snapshot

### UFS snapshot
0    1      *       *       *       root    /usr/local/sbin/snapshot make -g7 /SRCPATH:Daily
0      2      *       *       1       root    /usr/local/sbin/snapshot make -g4 /SRCPATH:Weekly

2. 掛載 UFS snapshot 的 shell script

#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        SSSRCPATH="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SnapshotName="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        MountPath="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 Snapshot_Src_Path SnapshotName MountPath"
        echo "Example: $0 /:Weekly.0 /mnt"
        exit 1
else
        echo "Try to mount $SSSRCPATH:$SnapshotName to $MountPath now."
fi

MC=`df -h |grep $MountPath |wc -l`

if [ -d $SSSRCPATH ]; then
        if [ -f $SSSRCPATH/.snap/$SnapshotName ]; then
                if [ -d $MountPath ]; then
                        if [ $MC -gt 0 ]; then
                                echo "MountPath $MountPath had been mounted!"
                        else
                                snapshot mount $SSSRCPATH:$SnapshotName $MountPath
                                df -h |grep $MountPath
                                echo "MountPath $MountPath was mounted."
                        fi
                else
                        echo "MountPath $MountPath does not existed!"
                fi
        else
                echo "Snapshot $SSSRCPATH/.snap/$SnapshotName does not existed!"
        fi
else
        echo "Snapshot source path $SSSRCPATH does not existed!"
fi

3. 掛載範例

# mkdir -p /SNAPSHOT/Mount/Path/Weekly0

/PATH/TO/mount_UFS_snapshot.sh /SRCPATH Weekly.0 /SNAPSHOT/Mount/Path/Weekly0

4. 卸載 UFS snapshot 的 shell script

#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        MountPath="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 MountPath"
        echo "Example: $0 /mnt"
        exit 1
else
        echo "Try to umount $MountPath now."
fi

MC=`df -h |grep $MountPath |wc -l`

if [ -d $MountPath ]; then
        if [ $MC -gt 0 ]; then
                snapshot umount $MountPath
                df -h |grep $MountPath
                echo "MountPath $MountPath had been umounted!"
        else
                echo "MountPath $MountPath had not been mounted!"
        fi
else
        echo "MountPath $MountPath does not existed!"
fi

5. 卸載範例

/PATH/TO/umount_UFS_snapshot.sh /SNAPSHOT/Mount/Path/Weekly0

6. UFS 做 snapshot 速度還蠻慢的. (相較於 ZFS 的快速完成 snapshot)

參考資料
FreeBSD UFS/ZFS Snapshot Management Environment

Saturday, March 03, 2012

在 FreeBSD 及 Ubuntu 啟動 MLS SSD 支援 TRIM

目前手邊的機器每一台都至少裝了一顆 MLC SSD. 不論是當系統碟或當暫存區使用, 都有需要啟動 TRIM. 剛好 FreeBSD 8.2 及 Ubuntu 11.10 的 kernel 都有支援 TRIM. 所以就參考文件設定.

[[[ FreeBSD 8.2 ]]]
1. Reboot and login in Single User Mode
2. tunefs -p /dev/adXXsXX
3. tunefs -t enable /dev/adXXsXX
4. Reboot the machine
5. Check the setting

HHHHH(XXXXXX)/root# tunefs -p /dev/adXsXX
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         enabled
tunefs: maximum blocks per file in a cylinder group: (-e)  2048
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

[[[ Ubuntu 11.10 ]]]
1. Edit /etc/fstab and add "discard,noatime,nodiratime,".

root@HOSTNAME:~# diff /etc/fstab.bak /etc/fstab
10c10
< UUID=dXXXdXXX-5899-XXXX-XXXd-XXXXcXXX8XXX /               ext4    errors=remount-ro 0       1
---
> UUID=dXXXdXXX-5899-XXXX-XXXd-XXXXcXXX8XXX /               ext4    discard,noatime,nodiratime,errors=remount-ro 0       1

2. Edit /etc/rc.local and add following statements before 'exit 0'.

### For tweak SSD
echo noop > /sys/block/sda/queue/scheduler
echo 1 > /sys/block/sda/queue/iosched/fifo_batch

3. Reboot the machine

4. 參考這篇做測試. HOWTO: Check If TRIM On Ext4 Is Enabled And Working On Ubuntu And Other Distributions

參考資料 Reference:
TRIM support on UFS2 FreeBSD 9
Racklin's 阿土伯程式大觀園: 為 Thinkpad T61 Debian Linux 換上 SSD
HOWTO: Configure Ext4 to Enable TRIM Support for SSDs on Ubuntu and Other Distributions
HOWTO: Check If TRIM On Ext4 Is Enabled And Working On Ubuntu And Other Distributions
How to Tweak Your SSD in Ubuntu for Better Performance

在 FreeBSD, PC-BSD, Ubuntu, Mint, CentOS, Fedora, OpenSUSE, Debian 上使用 tmpfs 掛載 /tmp

首先說明一下, 以下是給懶人專用, RAM Disk 大小都是預設最大到 50% 記憶體. 如果有需要調整空間使用上限或在大型伺服器上實做, 請務必研究一下附件還有 tmpfs. 另外 NetBSD 5, OpenSolaris 及 Indiana 就不用傷腦筋了, 系統預設已經使用 tmpfs(在 NetBSD 使用 mfs) 掛載 /tmp.

由於現代的機器的記憶體基本上都是 GB 等級. 讓記憶體閒著也是一種浪費. 而更重要的原因是不想讓讀寫 /tmp 的動作大量存取 MLC SSD（雖然已經啟動了 TRIM）. 再加上 guest VM 常常讀寫, 多跑幾台對 VM host 還是有效能上的影響. 就這樣看著大家的文件跟著做. 也順手將手邊的測試用 guest VM 加上這個簡單的設定. 以下就是實際上在各平台上的設定方式:

[[[ FreeBSD 8, PC-BSD 8 & 9 beta ]]]
1. Edit /etc/fstab

tmpfs  /tmp  tmpfs  rw 0 0

2. Reboot the machine

[[[ Ubuntu 11.X 12.04 LTS, Mint 11 ]]]
1. Edit /etc/init.d/ramtmp.sh

#!/bin/sh
# RamDisk tmp
PATH=/sbin:/bin:/usr/bin:/usr/sbin

mkdir /dev/shm/tmp
chmod 1777 /dev/shm/tmp
mount --bind /dev/shm/tmp /tmp

2. chmod 755 /etc/init.d/ramtmp.sh
3. cd /etc/rcS.d; ln -s /etc/init.d/ramtmp.sh SXXramtmp
XX : 必須在所有需要 /tmp 目錄的服務之前. 請注意英文字母排序.
簡單的作法就是在 /etc/rcS.d 下 grep tmp ./* 看有哪些檔案需要 /tmp 目錄.
4. Reboot the machine

[[[ CentOS 6, Fedora 15, ScientificLinux 5.6 ]]]
1. Edit /etc/rc.d/init.d/ramtmp

#!/bin/bash
# chkconfig: 35 80 70
# description: RAMDisk4Tmp
#
# History:
### 2012/02/25 RamDisk tmp

PATH=/sbin:/bin:/usr/bin:/usr/sbin

mkdir /dev/shm/tmp
chmod 1777 /dev/shm/tmp
mount --bind /dev/shm/tmp /tmp

2. chmod 755 /etc/rc.d/init.d/ramtmp
3. chkconfig --add ramtmp
4. Reboot the machine

[[[ OpenSUSE 11.4 ]]]
1. Edit /etc/fstab

tmpfs  /tmp  tmpfs  defaults 0 0

2. Reboot the machine

[[[ Smoothwall express 3.0 sp2 ]]]
1. Edit /etc/fstab

none  /tmp  tmpfs  defaults 0 0

2. Reboot the machine

[[[ ArchLinux 3.0 ]]]
1. Edit /etc/fstab

tmpfs  /tmp  tmpfs  nodev,nosuid 0 0

2. Reboot the machine

[[[ Debian 6 ]]]
1. Edit /etc/fstab

tmpfs /tmp tmpfs defaults,nodev,nosuid 0 0

2. Reboot the machine

[[[ Windows 7 Ultimate, Windows server 2003, Windows XP ]]]
請安裝 Dataram_RAMDisk_V3.5.130R22, 詳細步驟請看參考資料.

參考資料 Reference:
FreeBSD : 【FreeBSD】用 tmpmfs 或 tmpfs 建立 Ramdisk
Ubuntu : Firefox & Chrome Cache on RAM Drive -Fedora / Ubuntu
在 Ubuntu 將 Firefox 網頁快取檔，設到 Ramdisk 的方法
CentOS : [筆記]Linux Tmpfs 系統加速實作
OpenSUSE : SDB:SSD performance
Debian : HOWTO: Using tmpfs for /tmp
Windows : [桌面系統]Windows 7 (32bit/64bit) 使用RamDisk 外加ReadyBoost 增加效能

Saturday, February 18, 2012

在 FreeBSD 使用 portmaster 更新已安裝的 ports - 程式碼修改

程式碼:

#!/usr/local/bin/bash

PKGLISTPATH="/PATH/TO/STORE/PKGLIST"

if [ -n "$1" ]; then
        keyword="$1"
        shift
else
        echo "Usage: $0 keyword"
        echo "Example: $0 lib"
        exit 1
fi

echo $keyword

if [ -f $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt ]
then
        echo "Had $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt"
else
        echo "Generating $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt"
        portversion -l '<' > $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt
fi

for i in $(cat $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt | grep $keyword | awk '{ print $1 }'); do portmaster -D $i; done

備份系統設定且紀錄差異

為了玩玩在 shell script 使用函式功能, 寫了一隻簡單的程式碼紀錄系統設定的差異. 不過增加項目還是需要修改程式碼. 暫時先能用再來看如何修改.
程式碼:

#!/usr/local/bin/bash

BasePath="/PATH/TO/BACKUP/DIRECTORY"
PATH=$PATH

BackupConfig(){
if [ -z $1 ]
# Checks if any params.
then
 echo "No parameters passed to function."
 return 0
else
 ProgName=$1
fi

if [ $2 ]
then
 BackupFolder=$2
fi

case $ProgName in
postfix) CMD="postconf -n"
  ;;
sysctl)  CMD="sysctl -a"
  ;;
pkglist) CMD="ls /var/db/pkg"
  ;;
*)  echo "Do not support this program $ProgName"
                ;;
esac

mkdir -p $BasePath/$BackupFolder
rm -f $BasePath/$BackupFolder/${ProgName}_old.txt

echo "Check if had OLD data : "
if [ -f $BasePath/$BackupFolder/${ProgName}_now.txt ]; then
 echo "   Had OLD data! Rename it to ${ProgName}_old.txt"
 mv $BasePath/$BackupFolder/${ProgName}_now.txt $BasePath/$BackupFolder/${ProgName}_old.txt
else
 echo "   Do not have OLD data."
fi

echo "Backup configuration of ${ProgName} now."
$CMD > $BasePath/$BackupFolder/${ProgName}_now.txt

echo "Generate diff log : "
if [ -f $BasePath/$BackupFolder/${ProgName}_old.txt ]; then
 echo "   Generate diff log between ${ProgName}_new.txt and ${ProgName}_old.txt."
 diff $BasePath/$BackupFolder/${ProgName}_now.txt $BasePath/$BackupFolder/${ProgName}_old.txt > $BasePath/$BackupFolder/diff-`date +"%Y%m%d"`.txt
 find  $BasePath/$BackupFolder/ -type f -size 0 | xargs rm -f
else
 echo "   Do not have OLD data."
fi

}


### Backup postfix setting
BackupConfig postfix postfix_config_backup

### Backup sysctl setting
BackupConfig sysctl sysctl_config_backup

### Backup /var/db/pkg
BackupConfig pkglist installed-pkglist

Monday, January 30, 2012

Shell script 搭配 ipfw table 阻擋入侵 sshd 攻擊 authlog_banip.sh 修正版

1. 目前已將之前只有阻擋 30 天改為先阻擋 30 天, 若是 30 ~ 90 天前仍有多筆紀錄就再加鎖一段時間直到符合 Good Boy 條件.

2. 程式碼如下:
For SSHD


#!/usr/local/bin/bash

LOGFILE="/PATH/TO/auth.log"
PROCFOLDER="/PATH/TO/auth_sshd_banip"
TOTALBANLOGFILE="/PATH/TO/auth_sshd_banip.log"
TOTALBANLOGFILESORTED="/PATH/TO/auth_sshd_banip_sorted.log"
TOTALBANLOGFILEFROMPROCFOLDERSORTED="/PATH/TO/auth_sshd_banip_from_procfolder_sorted.log"
TOTALRESCUELIST="/PATH/TO/auth_sshd_rescue_list.log"
CRONJOBDATE="crontjobdate.log"
TODAYLOGFM=`date +"%b %e"`
DefaultCheckPeriod="30"
MVFilePeriod="90"

TODAY=`date +"%Y%m%d"`
OLDDAY=`date -v-${DefaultCheckPeriod}d +"%Y%m%d"`
OLDYEAR=`date -v-${DefaultCheckPeriod}d +"%Y"`

fwcmd="/sbin/ipfw"

echo "Prepare folder and file"
mkdir -p $PROCFOLDER
touch $TOTALBANLOGFILE
touch $TOTALRESCUELIST
touch $PROCFOLDER/$CRONJOBDATE
touch $TOTALBANLOGFILEFROMPROCFOLDERSORTED

SSHDErrorCounter1=`cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Invalid user" | awk '{ print $10 }' | wc -l`
SSHDErrorCounter2=`cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Did not receive identification string from" | awk '{ print $12 }' | wc -l`
SSHDErrorCounter3=`cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "not allowed because none of user" | awk '{ print $9 }' | wc -l`

if [ $SSHDErrorCounter1 -ne 0 ]||[ $SSHDErrorCounter2 -ne 0 ]||[ $SSHDErrorCounter3 -ne 0 ]||[ -f $PROCFOLDER/sshdpre-$TODAY.log ]; then
        echo "Write pre-process file"
        touch $PROCFOLDER/sshdpre-$TODAY.log
        cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Invalid user" | awk '{ print $10 }' | sort -n | sort -u >> $PROCFOLDER/sshdpre-$TODAY.log
        cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Did not receive identification string from" | awk '{ print $12 }' | sort -n | sort -u >> $PROCFOLDER/sshdpre-$TODAY.log
        cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "not allowed because none of user" | awk '{ print $9 }' | sort -n | sort -u >> $PROCFOLDER/sshdpre-$TODAY.log
        cp -a $PROCFOLDER/sshdpre-$TODAY.log /tmp
        cat /tmp/sshdpre-$TODAY.log | sort -n | sort -u > $PROCFOLDER/sshdpre-$TODAY.log
        rm -f /tmp/sshdpre-$TODAY.log

        echo "Collect new ban IP"
        for ip in $(awk '{ print }' $PROCFOLDER/sshdpre-$TODAY.log)
        do
                if [ `grep $ip $PROCFOLDER/sshd-*.log | wc -l` -gt 0 ]; then
                        echo $ip "Old bad boy"
                else
                        echo $ip
                        
                        touch $PROCFOLDER/sshd-$TODAY.log
                        echo $ip >> $PROCFOLDER/sshd-$TODAY.log
                        echo $ip >> $TOTALBANLOGFILE
                fi
        done

        echo "Ban bad boy"
        if [ -f $PROCFOLDER/sshd-$TODAY.log ]; then
                for banip in $(awk '{ print }' $PROCFOLDER/sshd-$TODAY.log)
                do
                        #Ban IP
                        echo "Check $banip"

                        if [ `ipfw table 3 list | grep $banip | wc -l` -gt 0 ]; then
                                echo "Had ban bad boy($banip)."
                        else
                                echo "Ban bad boy($banip) now."
                                ${fwcmd} table 3 add $banip/32
                                ${fwcmd} table 4 add $banip/32
                        fi
                done
        else
                echo "No bad boy"
        fi

        echo "Rescue good boy"
        if [ -f $PROCFOLDER/sshd-$OLDDAY.log ]; then
                for rescueip in $(awk '{ print }' $PROCFOLDER/sshd-$OLDDAY.log)
                do
                        #Rescue IP
                        echo "Check $rescueip"
                        
                        BadBoyDCPCounter=`find $PROCFOLDER -d 1 -ctime +${DefaultCheckPeriod} -type f -name 'sshd*.log' -exec grep $rescueip {} \; | wc -l`
                        BadBoyMVFCounter=`find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'sshd*.log' -exec grep $rescueip {} \; | wc -l`

                        if [ $BadBoyDCPCounter -lt 1 ]; then
                                if [ $BadBoyMVFCounter -le 2 ]; then
                                        echo "Rescue $rescueip"
                                        ${fwcmd} table 3 delete $rescueip/32
                                        ${fwcmd} table 4 delete $rescueip/32
                                
                                        grep -v $rescueip $TOTALBANLOGFILE > /tmp/stillbanip-$TODAY
                                        cp -f /tmp/stillbanip-$TODAY $TOTALBANLOGFILE
                                        rm -f /tmp/stillbanip-$TODAY
                                        echo $rescueip >> $TOTALRESCUELIST
                                        echo "Rescued"
                                else
                                        echo "Not bad boy in ${DefaultCheckPeriod} days, but had bad boy record in ${MVFilePeriod} days."
                                fi
                        else
                                echo "Still bad boy in ${DefaultCheckPeriod} days"
                        fi
                done
                
                #Move to old folder
                echo "Move old file to storage folder"
                mkdir -p $PROCFOLDER/$OLDYEAR
                mv $PROCFOLDER/sshd*-$OLDDAY.log $PROCFOLDER/$OLDYEAR
        else
                echo "No candidate file"
        fi

        cat $TOTALBANLOGFILE | sort -n | sort -u > $TOTALBANLOGFILESORTED
        
else
        echo "No bad boy"
fi

### Move old files to backup folder
find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'sshd*.log' -exec mv {} $PROCFOLDER/$OLDYEAR \;
cat $PROCFOLDER/sshd-*.log | sort -n | sort -u > $TOTALBANLOGFILEFROMPROCFOLDERSORTED

### Additional rescure goodboy
for rescueip in `diff $TOTALBANLOGFILEFROMPROCFOLDERSORTED $TOTALBANLOGFILESORTED | grep -v a | awk '{ print $2 }' | sort -r -n`
do
        echo "Additional rescure goodboy from diff $TOTALBANLOGFILEFROMPROCFOLDERSORTED and $TOTALBANLOGFILESORTED"     
        echo "Rescue $rescueip"
        ${fwcmd} table 3 delete $rescueip/32
        ${fwcmd} table 4 delete $rescueip/32

        grep -v $rescueip $TOTALBANLOGFILE > /tmp/stillbanip-$TODAY
        cp -f /tmp/stillbanip-$TODAY $TOTALBANLOGFILE
        rm -f /tmp/stillbanip-$TODAY
        echo $rescueip >> $TOTALRESCUELIST
        echo "Rescued"
done

cat $TOTALBANLOGFILE | sort -n | sort -u > $TOTALBANLOGFILESORTED

echo $TODAY > $PROCFOLDER/$CRONJOBDATE

### For backup and administration
/PATH/TO/BACKUP_SCRIPT

For MAIL


#!/usr/local/bin/bash

PATH=$PATH

LOGFILE="/PATH/TO/maillog"
DOVECOTERRORLOGFILE="/PATH/TO/dovecot/error.log"
DOVECOTINFOLOGFILE="/PATH/TO/dovecot/info.log"
PROCFOLDER="/PATH/TO/mail_banip"
DAILYLOG="mail_banip"`date +"%Y%m%d"`".log"
TMPFILE="/tmp/mailban-"`date +"%s"`
TODAYLOGFM=`date +"%b %e"`
fwcmd="/sbin/ipfw"

DefaultCheckPeriod="30"
MVFilePeriod="30"

OLDDAILYLOG="mail_banip"`date -v-${DefaultCheckPeriod}d +"%Y%m%d"`".log"
OLDYEAR=`date -v-${MVFilePeriod}d +"%Y"`

mkdir -p $PROCFOLDER

MAILErrorString1="NOQUEUE: reject"
MAILErrorCounter1=`cat $LOGFILE | grep "$TODAYLOGFM" | grep postfix | grep "$MAILErrorString1" | wc -l`
MAILErrorString2="auth-worker(default): Error: pam"
MAILErrorCounter2=`cat $DOVECOTERRORLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString2" | awk 'BEGIN { FS = ": " } ; { print $3 }' | awk 'BEGIN { FS = "," } ; { print $2 }' | sort -n | sort -u | sed -Ee 's/)//' | wc -l`
MAILErrorString3="pop3-login: Info: Aborted login (auth failed"
MAILErrorCounter3=`cat $DOVECOTINFOLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString3" | awk 'BEGIN { FS = ", " } ; { print $4 }' | awk 'BEGIN { FS = "=" } ; { print $2 }' | sort -n | sort -u | wc -l`

if [ $MAILErrorCounter1 -ne 0 ]||[ $MAILErrorCounter2 -ne 0 ]||[ $MAILErrorCounter3 -ne 0 ]||[ -f $PROCFOLDER/$DAILYLOG ]; then
        if [ ! -f $PROCFOLDER/$DAILYLOG ]; then
                touch $PROCFOLDER/$DAILYLOG
        fi

        echo "Collect Bad boy"
        touch $TMPFILE
        cat $LOGFILE | grep "$TODAYLOGFM" | grep postfix | grep "NOQUEUE: reject" | awk 'BEGIN { FS = "[" } ; { print $3 }' | awk 'BEGIN { FS = "]" } ; { print $1 }' | sort -n | sort -u >> $PROCFOLDER/$DAILYLOG
        cat $DOVECOTERRORLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString2" | awk 'BEGIN { FS = ": " } ; { print $3 }' | awk 'BEGIN { FS = "," } ; { print $2 }' | sort -n | sort -u | sed -Ee 's/)//' >> $PROCFOLDER/$DAILYLOG
        cat $DOVECOTINFOLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString3" | awk 'BEGIN { FS = ", " } ; { print $4 }' | awk 'BEGIN { FS = "=" } ; { print $2 }' | sort -n | sort -u >> $PROCFOLDER/$DAILYLOG
        cat $PROCFOLDER/$DAILYLOG | sort -n | sort -u | grep -v "192.168.58" | grep -v "192.168.68" | grep -v "192.168.8" | grep -v "60.248.16.64" | grep -v "60.248.16.65" | grep -v "60.248.16.66" > $TMPFILE
        cp -f $TMPFILE $PROCFOLDER/$DAILYLOG

        echo "Ban Bad Boy"
        for banip in $(awk '{ print }' $PROCFOLDER/$DAILYLOG)
        do
                if [ `ipfw table 3 list | grep $banip | wc -l` -gt 0 ]; then
                        echo "Had ban bad boy($banip)."
                else
                        echo "Ban bad boy($banip) now."
                        ${fwcmd} table 3 add $banip/32
                        ${fwcmd} table 4 add $banip/32
                fi
        done
        
        rm -f $TMPFILE
fi

if [ -f $PROCFOLDER/$OLDDAILYLOG ]; then
        for rescueip in $(awk '{ print }' $PROCFOLDER/$OLDDAILYLOG)
        do
                echo "Rescue $rescueip"
                ${fwcmd} table 3 delete $rescueip/32
                ${fwcmd} table 4 delete $rescueip/32
        done
else
        echo "No candidate file"
fi

#Move to old folder
echo "Move old file to storage folder"
mkdir -p $PROCFOLDER/$OLDYEAR
find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'mail_banip*.log' -exec mv {} $PROCFOLDER/$OLDYEAR \;

For HTTPD


#!/usr/local/bin/bash

PATH=$PATH

LOGFILE="/PATH/TO/httpd-access.log"
PROCFOLDER="/PATH/TO/httpd_banip"
DAILYLOG="httpd_banip"`date +"%Y%m%d"`".log"
TMPFILE="/tmp/httpdban-"`date +"%s"`
TODAYLOGFM=`date +"%d/%b/%Y"`
fwcmd="/sbin/ipfw"

DefaultCheckPeriod="30"
MVFilePeriod="30"

OLDDAILYLOG="httpd_banip"`date -v-${DefaultCheckPeriod}d +"%Y%m%d"`".log"
OLDYEAR=`date -v-${MVFilePeriod}d +"%Y"`

mkdir -p $PROCFOLDER

ErrorString1="xmlrpc.php"
ErrorCounter1=`cat $LOGFILE | grep "$TODAYLOGFM" | grep "$ErrorString1" | grep 404 | awk 'BEGIN { FS = " " } ; { print $1 }' | grep -v "192.168.58" | grep -v "192.168.68" | grep -v "192.168.8" | grep -v "60.248.16.64" | grep -v "60.248.16.65" | grep -v "60.248.16.66" | wc -l`

if [ $ErrorCounter1 -ne 0 ]||[ -f $PROCFOLDER/$DAILYLOG ]; then
        if [ ! -f $PROCFOLDER/$DAILYLOG ]; then
                touch $PROCFOLDER/$DAILYLOG
        fi

        echo "Collect Bad boy"
        touch $TMPFILE
        cat $LOGFILE | grep "$TODAYLOGFM" | grep "$ErrorString1" | grep 404 | awk 'BEGIN { FS = " " } ; { print $1 }' | sort -n | sort -u >> $PROCFOLDER/$DAILYLOG
        cat $PROCFOLDER/$DAILYLOG | sort -n | sort -u | grep -v "192.168.58" | grep -v "192.168.68" | grep -v "192.168.8" | grep -v "60.248.16.64" | grep -v "60.248.16.65" | grep -v "60.248.16.66" > $TMPFILE
        cp -f $TMPFILE $PROCFOLDER/$DAILYLOG
        
        echo "Ban Bad Boy"
        for banip in $(awk '{ print }' $PROCFOLDER/$DAILYLOG)
        do
                if [ `ipfw table 3 list | grep $banip | wc -l` -gt 0 ]; then
                        echo "Had ban bad boy($banip)."
                else
                        echo "Ban bad boy($banip) now."
                        ${fwcmd} table 3 add $banip/32
                        ${fwcmd} table 4 add $banip/32
                fi
        done
        
        rm -f $TMPFILE
fi

if [ -f $PROCFOLDER/$OLDDAILYLOG ]; then
        for rescueip in $(awk '{ print }' $PROCFOLDER/$OLDDAILYLOG)
        do
                echo "Rescue $rescueip"
                ${fwcmd} table 3 delete $rescueip/32
                ${fwcmd} table 4 delete $rescueip/32
        done
else
        echo "No candidate file"
fi

#Move to old folder
echo "Move old file to storage folder"
mkdir -p $PROCFOLDER/$OLDYEAR
find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'httpd_banip*.log' -exec mv {} $PROCFOLDER/$OLDYEAR \;

3. 以上程式碼為自由軟體, 請自行取用. 強烈建議服用前先看清楚再服用, 避免發生意外.

Friday, September 30, 2011

如何掛載 FreeBSD 光碟影像檔

Script 如下 :

mail(xxx)# more mount_FreeBSDDVD.sh
#!/bin/sh

mkdir -p /FreeBSD-RELEASEDVD
mdconfig -a -t vnode -f /PATH/TO/FreeBSD-IMAGE.iso -u 1
mount -t cd9660 /dev/md1 /FreeBSD-RELEASEDVD

mail(xxx)# more umount_FreeBSDDVD.sh
#!/bin/sh

umount /FreeBSD-RELEASEDVD
mdconfig -d -u 1

Jengyic's 正義部落