Jengyic's 正義部落: shell script

Showing posts with label shell script. Show all posts

Sunday, August 03, 2014

掃描子網路下的主機是否有開啟服務埠口

這個計畫的目的是找出子網路下有哪些主機提供服務, 目前只有簡單的偵測 Ping(ICMP), Web(80), DNS(open resolver), SSH(22), SMTP(25). 未來將陸續增加偵測服務的項目. 雖然已經有 nmap 等優秀軟體, 不過仍舊希望能發展出簡易的程式, 更希望能匯聚更多偵測點讓資料搜集時間縮短. 接下來將開始設計資料庫結構及提供查詢的API.

目前尚有資料來源的問題, 就是各國網段清單. 目前是從 ip2nation 計算出網段. 但是這個資料並不是即時資料, 所以如果有哪位大德知道去哪找免費的清單, 請不吝告訴在下.

專案網址: https://github.com/jengyic/IP_Research

Saturday, October 19, 2013

目錄下大量檔案 piconv 轉碼的簡易 shell script

之前為了將一些 Big-5 資料庫檔案轉成 UTF-8 用過 iconv 但是遇到某些字元就會出錯無法完成翻譯。後來找到 piconv 可以完整將所有資料庫匯出檔案轉碼。今天為了看 SGU Season 1 找到簡體字幕，沒想到用 piconv 一樣沒出錯完成轉碼。

Google雲端硬碟公開分享檔名 piconvTrans.sh

Sunday, December 30, 2012

使用 shell script 透過 SSH 在遠端伺服器執行程式

這隻程式只是個簡單的實做連線前偵測遠端伺服器是否可以 ping 到及服務埠是否有開啟. 程式碼如下

FreeBSD下用 shell script 建立ZFS snapshot 更新版

此次更新的目的是讓 ZFS 不使用 ls /PATH/.zfs/snapshot 目錄判斷快照數量(為了避開 Bad file descriptor 問題). 中間版本曾經使用 /usr/local/bin/snapshot list 進行, 但是速度較慢. 好在這個檔案是 shell script. 看了一下 op_list 函式的內容看到取得目前已有的快照方法, 也了解到這隻程式慢在還要查詢及計算使用空間等數據. 所以將查詢改用 zfs list -H -t snapshot -o name | grep "ZPOOL_NAME/ZVOL_NAME@". (ZPOOL_NAME/ZVOL_NAME 可以使用 zfs list -H -o name ${volpath} 查詢得到)

使用 SSHFS 掛載遠端目錄

1. 安裝 SSHFS
FreeBSD 下請安裝 /usr/ports/sysutils/fusefs-sshfs
Ubuntu/Debian 下請 apt-get install sshfs

Mount :
sshfs RemoteServerAccountName@SSHServerName:/Remote/path /Mount/path -o follow_symlinks

Umount :
FreeBSD : umount /Mount/path
Ubuntu/Debian : fusermount -u /Mount/path

2. 使用掛載 shell script
目前暫時不做作業系統版本判斷. 所以有以下的兩組 shell script:

讓 cron 可以每隔幾秒執行 shell script

之前使用每五分鐘檢查是否有嘗試入侵的機器, 發現有些機器會一下送來一堆連線. 所以只好將執行時間降到以秒為單位. 由於檢查用的 shell script 約在幾秒內完成, 所以就用下列的 shell script 來達成.

C範例程式 : 判斷 IP 是否在給定的 subnet

為了搭配自己的 shell script 寫了這隻小程式, 沒有做一些多餘的錯誤檢查(因為 shell script 餵過去的資料已經處理過).只支援 IPv4 及一種 subnet 表示法. (只用了 stdio.h stdlib.h string.h )
程式碼

#include 
#include 
#include 

int main(int argc, char **argv)
{
 int uip[4], dst[4], nm[4], i = 0, netmask;
 char *delim = ".";
 char *dstdelim = "/";
 char * pch;
 char * dstip;
 char * mask;
 int j = 128;

 /* Check number of argc */
 if(argc != 3){
  printf("Usage : ipinsubnet UIP DSTIP/NETMASK\n");
  printf("Example : ipinsubnet 192.168.0.1 192.168.0.0/24\n");
  return 1;
 }

 /* For uip */
 i = 0;
 pch = strtok(argv[1], delim);
 while (pch != NULL){
  uip[i] = atoi(pch);
  pch = strtok(NULL, delim);
  i++;
 } 

 /* For dst IP and netmask */
 i = 0;
 /* Get dst IP */
 dstip = strtok(argv[2], dstdelim);

 /* Get netmask */
 mask = strtok(NULL, dstdelim);
 netmask = atoi(mask);

        pch = strtok(dstip, delim);
 while (pch != NULL){
  dst[i] = atoi(pch);
  pch = strtok(NULL, delim);
  i++;
 }

 /* Generate netmask */
 i = 0;
 while((netmask - 8) >= 0){
  nm[i] = 255;
  netmask-=8;
  i++;
 }

 if( netmask > 0 ){
  nm[i] = 0;
  for(; netmask > 0; netmask--){ nm[i] = nm[i] | j; j >>= 1; }
  i++;
 }
 for(;i < 4; i++){ nm[i] = 0; }

 /* Compare uip and dst with netmask */
 for(i = 0; i < 4; i++){
  if( (uip[i] & nm[i]) != (dst[i] & nm[i]) ){ printf("F\n"); return 0; }
 }
 printf("T\n");

 return 0;
}

使用範例

$ ./ipinsubnet
Usage : ipinsubnet UIP DSTIP/NETMASK
Example : ipinsubnet 192.168.0.1 192.168.0.0/24

$ ./ipinsubnet 192.168.68.1 192.168.78.0/16
T
$ ./ipinsubnet 192.168.68.1 192.168.78.0/20
T
$ ./ipinsubnet 192.168.68.1 192.168.78.0/21
F
$ ./ipinsubnet 192.168.68.1 192.168.78.0/22
F

Saturday, September 15, 2012

自家用 ipfw log 統計

雖然有 security/ipfwcount 好用, 但是總想可以自由的分析 ipfw.log. 所以就寫了以下的 stupid script. 哈哈 ... 但是它可以用.

#!/usr/local/bin/bash
##
##  FreeBSD ipfw log analyzer
##

PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin:$PATH"

TS=`date +"%s"`
TMPFILE="/tmp/ana_ipfw-$TS"
TMPFILEsort="/tmp/ana_ipfw_sort-$TS"
LC_ALL=C
LANG=C

### Default options
datelimit=no
portlimit=no
iplimit=no
verbose=no
sorted=no
onlydata=no
datestr=""
logfile="/var/log/ipfw.log"

### Help
usage () {
 echo "Usage: $0 [-p] [-i] [-v] [-s] [-o] [-d date_limit] [filename]"
 echo "   -p  port analysis"
 echo "   -i  IP address analysis"
 echo "   -d  date format"
 echo "   +DDAY   from DDAY to today"
 echo "   +-DDAY  DDAY days ago"
 echo "   YYYYMMDD-YYYYMMDD  from first YYYYMMDD to last YYYYMMDD"
 echo "   total   all data records in the log file"
 echo "   -s  sorted by count"
 echo "   -o  only data and disable verbose"
 echo "   -v  enable verbose messages"
 echo "   filename default : /var/log/ipfw.log"
}
if [ $# -eq 0 ]; then
 usage
 exit 1
fi

### Get arguments
args=`getopt dpivso $*`

if [ $? != 0 ]; then
 echo "$0:ERROR"
 exit 2
fi

set -- $args

for arg
do
 case "$arg" in
  -d )  datelimit=yes; shift ;;
  -p )  portlimit=yes; shift ;;
  -i )  iplimit=yes; shift ;;
  -v )  verbose=yes; shift ;;
  -s )  sorted=yes; shift ;;
  -o )  onlydata=yes; shift ;;
  -- )  datestr=$2; shift; shift ;;
 esac
done

if [ $datelimit == "no" ]; then
 if [ ! -z $datestr ]&&[ -f $datestr ]; then
  logfile=$datestr
 fi
else
 if [ ! -z $datestr ]&&[ ! -z $1 ]; then
  logfile=$1
 fi
fi

if [ ! -f $logfile ]; then
 echo "Error : Log file $logfile does not existed."
 exit 1
fi

if [ $onlydata == "yes" ]; then
 verbose=no
fi

### For debug
if [ $verbose == "yes" ]; then
 echo "Parameters"
 echo $args
 echo "Date $datelimit($datestr) Port $portlimit IP $iplimit Verbose $verbose Sort $sorted Logfile $logfile"
 echo ""
fi

### Generate date limitation
if [ $datelimit == "no" ]; then
 dtmp="0"
 datearray=($dtmp)
else
 if [ ! -z `echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }'` ]; then
  if [ ! -z `echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }' | awk 'BEGIN { FS = "-" } ; { print $2 }'` ]; then
   datearray=(`echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }' | awk 'BEGIN { FS = "-" } ; { print $2 }'`)
  else
   i=`echo $datestr | awk 'BEGIN { FS = "+" } ; { print $2 }'`
   dtmp=""
   for((; i >= 0; i--))
   do
    dtmp="$dtmp $i"
   done
   datearray=($dtmp)
  fi
 elif [ "$datestr" == "total" ]; then
  datestr="total"
 elif [ -z $datestr ]; then
  dtmp="0"
  datearray=($dtmp)
 else
  if [[ $datestr != *[!0-9\-]* ]]; then
   bdate=`echo $datestr | awk 'BEGIN { FS = "-" } ; { print $1 }'`
   edate=`echo $datestr | awk 'BEGIN { FS = "-" } ; { print $2 }'`
  else
   bdate=`date +"%Y%m%d"`
   edate=`date +"%Y%m%d"`
  fi

  todayts=`date -j "+%s"`
  bdatets=`date -j -f "%Y%m%d" "$bdate" "+%s"`
  edatets=`date -j -f "%Y%m%d" "$edate" "+%s"`

  if [ $bdatets -gt $todayts ]; then
   echo "Error : $bdate is future. I can not analyze it."
   exit 1
  else
   ib=`expr \( $todayts - $bdatets \) / 86400`
  fi
  if [ $edatets -gt $todayts ]; then
   if [ $verbose == "yes" ]; then
    echo "Warning : $edate is future. But I can replace it with today("`date +"%Y%m%d"`")."
    echo ""
   fi
   ie=0
  else
   ie=`expr \( $todayts - $edatets \) / 86400`
  fi

  dtmp=""
  for((i = $ib; i >= $ie; i--))
  do
   dtmp="$dtmp $i"
  done
  datearray=($dtmp)
 fi
fi

### Create tmp file
if [ $verbose == "yes" ]; then
 echo "Create $TMPFILE"
 echo ""
fi
touch $TMPFILE

### Get log
if [ $verbose == "yes" ]; then
 echo "Get log from $logfile"
fi
if [ $datelimit == "no" ]; then
 if [ $verbose == "yes" ]; then
  date +"%b %e"
 fi
 keyfilter=`date +"%b %e"`
 grep "$keyfilter" $logfile >> $TMPFILE
elif [ "$datestr" == "total" ]; then
 cat $logfile > $TMPFILE
else
 for dnum in ${datearray[@]}
 do
  if [ $verbose == "yes" ]; then
   date -v-${dnum}d +"%b %e"
  fi
  keyfilter=`date -v-${dnum}d +"%b %e"`
  grep "$keyfilter" $logfile >> $TMPFILE
 done
fi
if [ $verbose == "yes" ]; then
 echo ""
fi

if [ $verbose == "yes" ]; then
 echo "Count lines in $TMPFILE"
 wc -l $TMPFILE
 echo ""
fi

### Port Analysis
if [ $portlimit == "yes" ]; then
 if [ $sorted == "yes" ]; then
  rm -f $TMPFILEsort
  touch $TMPFILEsort
 fi

 if [ $onlydata == "no" ]; then
  echo "[[[ Port Analysis ]]]"
  printf "%12s\t%12s\n" "Port Number" "Count"
  printf "%12s\t%12s\n" "------------" "--------"
 fi
 for portnum in `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { if( $11=="\-\>" ){ split($12, p, ","); print p[1] } else { print $11 } }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $2 }' | sort -u -n`
 do
  if [ $sorted == "yes" ]; then
   printf "%12s\t%12s\n" "$portnum" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { if( $11=="\-\>" ){ split($12, p, ","); print p[1] } else { print $11 } }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $2 }' | grep $portnum | wc -l` >> $TMPFILEsort
  else
   printf "%12s\t%12s\n" "$portnum" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { if( $11=="\-\>" ){ split($12, p, ","); print p[1] } else { print $11 } }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $2 }' | grep $portnum | wc -l`
  fi
 done
 if [ $sorted == "yes" ]; then
  cat $TMPFILEsort | awk 'BEGIN { FS = " " } ; { printf ( "%12s\t%12s\n", $1, $2) }' | sort -rgk2 > /tmp/ttt-$TS
  cat /tmp/ttt-$TS > $TMPFILEsort
  rm -f /tmp/ttt-$TS
  cat $TMPFILEsort
 fi
fi

### Client IP Analysis
if [ $iplimit == "yes" ]; then
 if [ $sorted == "yes" ]; then
  rm -f $TMPFILEsort
  touch $TMPFILEsort
 fi

 if [ $onlydata == "no" ]; then
  echo "[[[ Client IP Analysis ]]]"
  printf "%16s\t%12s\n" "IP Address" "Count"
  printf "%16s\t%12s\n" "----------------" "--------"
 fi
 for ipaddr in `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { print $10 }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $1 }' | sort -u -n`
 do
  if [ $sorted == "yes" ]; then
   printf "%16s\t%12s\n" "$ipaddr" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { print $10 }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $1 }' | grep $ipaddr | wc -l` >> $TMPFILEsort
  else
   printf "%16s\t%12s\n" "$ipaddr" `cat $TMPFILE | awk 'BEGIN { FS = " " } ; { print $10 }' | grep ":" | awk 'BEGIN { FS = ":" } ; { print $1 }' | grep $ipaddr | wc -l`
  fi
 done
 if [ $sorted == "yes" ]; then
  cat $TMPFILEsort | awk 'BEGIN { FS = " " } ; { printf ( "%16s\t%12s\n", $1, $2) }' | sort -rgk2 > /tmp/ttt-$TS
  cat /tmp/ttt-$TS > $TMPFILEsort
  rm -f /tmp/ttt-$TS
  cat $TMPFILEsort
 fi
fi

### Remove tmp file
if [ $verbose == "yes" ]; then
 echo ""
 echo "Remove $TMPFILE"
fi
rm -f $TMPFILE
if [ $sorted == "yes" ]; then
 if [ $verbose == "yes" ]; then
  echo "Remove $TMPFILEsort"
 fi
 rm -f $TMPFILEsort
fi

使用方法如下 :

Usage: /Path/to/ana_ipfw.sh [-p] [-i] [-v] [-s] [-o] [-d date_limit]  [filename]
   -p           port analysis
   -i           IP address analysis
   -d           date format
                        +DDAY   from DDAY to today
                        +-DDAY  DDAY days ago
                        YYYYMMDD-YYYYMMDD  from first YYYYMMDD to last YYYYMMDD
                        total   all data records in the log file
   -s           sorted by count
   -o           only data and disable verbose
   -v           enable verbose messages
   filename     default : /var/log/ipfw.log

分析範例如下 :
今天的 port 分析

# ./ana_ipfw.sh -p
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
          21               4
          22              14
          25              13
          80              30
         111               2
         199               2
         389               2
         443               8
         873               2
        2049               2
        3000              36
        3306              11
        5800               2
        5900               5
        6000               2
        6001               2

今天的 port 分析(按照數量排序)

# ./ana_ipfw.sh -ps
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
        3000              36
          80              30
          22              14
          25              13
        3306              12
         443               8
        5900               5
          21               4
        6001               2
        6000               2
        5800               2
        2049               2
         873               2
         389               2
         199               2
         111               2

最近三天的 port 分析

# ./ana_ipfw.sh -p -d +3
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
          21               7
          22              20
          25              41
          80              72
         111               2
         199               2
         389               2
         443              14
         873               2
        2049               2
        3000              72
        3306              57
        5800               2
        5900              12
        5901               3
        6000               7
        6001               2

最近三天的 port 分析(按照數量排序)

# ./ana_ipfw.sh -p -d +3 -s
[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
        3000              72
          80              72
        3306              57
          25              41
          22              20
         443              14
        5900              12
        6000               7
          21               7
        5901               3
        6001               2
        5800               2
        2049               2
         873               2
         389               2
         199               2
         111               2

最後開進工程模式(用來修 script 用, 應該還有 bug.).

# ./ana_ipfw.sh -pv
Parameters
-p -v --
Date no() Port yes IP no Verbose yes Sort no Logfile /var/log/ipfw.log

Create /tmp/ana_ipfw-1347716436

Get log from /var/log/ipfw.log
Sep 15

Count lines in /tmp/ana_ipfw-1347716436
     495 /tmp/ana_ipfw-1347716436

[[[ Port Analysis ]]]
 Port Number           Count
------------        --------
          21               4
          22              14
          25              13
          80              30
         111               2
         199               2
         389               2
         443               8
         873               2
        2049               2
        3000              36
        3306              12
        5800               2
        5900               5
        6000               2
        6001               2

Remove /tmp/ana_ipfw-1347716436

Wednesday, March 28, 2012

使用 rsnapshot 備份及比較設定檔

1. 安裝 rsnapshot
在 Ubuntu 下直接用 apt-get install rsnapshot.
在 FreeBSD 下請到 /usr/ports/sysutils/rsnapshot 下 make install clean. (在我的 FreeBSD 實體及虛擬環境中並沒有實際安裝測試, 以下僅用 Ubuntu 作為範例.)

2. 調整設定檔 /etc/rsnapshot.conf

# diff /etc/rsnapshot.conf /etc/rsnapshot.conf.20120322
27c27
< snapshot_root /.rsnapshot/
---
> snapshot_root /var/cache/rsnapshot/
97c97
< retain                hourly  8
---
> retain                hourly  6
100c100
< retain                monthly 6
---
> #retain       monthly 3
124c124
< logfile       /var/log/rsnapshot.log
---
> #logfile      /var/log/rsnapshot.log
230c230
< #backup       /home/          localhost/
---
> backup        /home/          localhost/
232,235c232
< backup        /usr/local/etc/ localhost/
< backup        /var/log/       localhost/
< backup        /var/mail/      localhost/
< backup        /root           localhost/
---
> backup        /usr/local/     localhost/

註 : 因為 /home 已將大部分資料放上 NFS, 已經有 ZFS snapshot. 就不浪費時間多做 snapshot.

3. 設定定期執行

0 */3 * * * /usr/bin/rsnapshot hourly
0 23 * * * /usr/bin/rsnapshot daily
15 23 * * 1 /usr/bin/rsnapshot weekly
30 23 1 * * /usr/bin/rsnapshot monthly
0 0 * * 1 /PATH/TO/REPORT/PROG/report_CHK_ETC_diff.sh

4. CHK_ETC_diff.sh

#!/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        CHKPATH="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SSKEYWORD="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SSBASE="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 check_path snapshot_keyword snapshot_base_path"
        echo "Example: $0 /etc Weekly /SNAPSHOT/System"
        exit 1
else
        echo "Collect difference between $CHKPATH and $SSBASE/${SSKEYWORD}*$CHKPATH."
fi

date +"%Y/%m/%d %H:%M:%S"
LAST_DIR="/"
CHKPATH=`echo $CHKPATH | sed -e 's/\///'`

for CONF in `find $LAST_DIR$CHKPATH -type f -name '*'`
do
        if [ `ls $SSBASE | grep ${SSKEYWORD} |wc -l` -lt 1 ]; then
                echo "$SSBASE/${SSKEYWORD}* does not existed!"
                exit 1
        fi
        ls -d $SSBASE/${SSKEYWORD}* | while read DIR
        do
                if [ $LAST_DIR = "/" ]; then
                        HOST=""
                else
                        HOST="/localhost"
                fi
                if [ -f ${LAST_DIR}${HOST}${CONF} ]&&[ -f ${DIR}/localhost${CONF} ]; then
                        if [ $LAST_DIR = "/" ]; then
                                LAST_DIR=""
                        fi
                        DIFFC=`diff ${LAST_DIR}${HOST}${CONF} ${DIR}/localhost${CONF} | wc -l`
                        if [ $DIFFC -gt 0 ]; then
                                echo "[[[ Difference between ${LAST_DIR}${HOST}${CONF} and ${DIR}/localhost${CONF} ]]]"
                                diff ${LAST_DIR}${HOST}${CONF} ${DIR}/localhost${CONF}
                                echo ""
                        fi
                        DIFFC=0
                else
                        if [ ! -f ${LAST_DIR}${HOST}${CONF} ]; then
                                echo "File ${LAST_DIR}${HOST}${CONF} does not existed."
                        fi
                fi

                LAST_DIR=$DIR
        done
done
echo ""
date +"%Y/%m/%d %H:%M:%S"

5. report_CHK_ETC_diff.sh

#!/bin/bash

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:${PATH}

CHKPROG="/PATH/TO/PROG/CHK_ETC_diff.sh /etc daily /.rsnapshot"
REPORTPATH="/PATH/TO/STORE/REPORT/report_CHK_ETC_diff"
REPORTKEYWORD="etcdiff"
REPORTTITLE="Check Difference of ETC Report"
REMAILADDR="ACCOUNT@DOMAIN.NAME"

mkdir -p $REPORTPATH/`date +"%Y"`
mv $REPORTPATH/${REPORTKEYWORD}_*.log $REPORTPATH/`date +"%Y"`
$CHKPROG > $REPORTPATH/${REPORTKEYWORD}_`hostname`_`date +"%Y%m%d"`.log
cat $REPORTPATH/${REPORTKEYWORD}_`hostname`_`date +"%Y%m%d"`.log | mail -s "$REPORTTITLE" $REMAILADDR

參考資料:
淺談linux下的快照備份軟體—rsnapshot

Sunday, March 18, 2012

在 FreeBSD 8 對 UFS filesystem 做 snapshot

1. 在 crontab 定期做 snapshot

### UFS snapshot
0    1      *       *       *       root    /usr/local/sbin/snapshot make -g7 /SRCPATH:Daily
0      2      *       *       1       root    /usr/local/sbin/snapshot make -g4 /SRCPATH:Weekly

2. 掛載 UFS snapshot 的 shell script

#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        SSSRCPATH="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        SnapshotName="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        MountPath="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 Snapshot_Src_Path SnapshotName MountPath"
        echo "Example: $0 /:Weekly.0 /mnt"
        exit 1
else
        echo "Try to mount $SSSRCPATH:$SnapshotName to $MountPath now."
fi

MC=`df -h |grep $MountPath |wc -l`

if [ -d $SSSRCPATH ]; then
        if [ -f $SSSRCPATH/.snap/$SnapshotName ]; then
                if [ -d $MountPath ]; then
                        if [ $MC -gt 0 ]; then
                                echo "MountPath $MountPath had been mounted!"
                        else
                                snapshot mount $SSSRCPATH:$SnapshotName $MountPath
                                df -h |grep $MountPath
                                echo "MountPath $MountPath was mounted."
                        fi
                else
                        echo "MountPath $MountPath does not existed!"
                fi
        else
                echo "Snapshot $SSSRCPATH/.snap/$SnapshotName does not existed!"
        fi
else
        echo "Snapshot source path $SSSRCPATH does not existed!"
fi

3. 掛載範例

# mkdir -p /SNAPSHOT/Mount/Path/Weekly0

/PATH/TO/mount_UFS_snapshot.sh /SRCPATH Weekly.0 /SNAPSHOT/Mount/Path/Weekly0

4. 卸載 UFS snapshot 的 shell script

#!/usr/local/bin/bash

PATH=$PATH
CHKPARA=0

if [ -n "$1" ]; then
        MountPath="$1"
        shift
else
        CHKPARA=1
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 MountPath"
        echo "Example: $0 /mnt"
        exit 1
else
        echo "Try to umount $MountPath now."
fi

MC=`df -h |grep $MountPath |wc -l`

if [ -d $MountPath ]; then
        if [ $MC -gt 0 ]; then
                snapshot umount $MountPath
                df -h |grep $MountPath
                echo "MountPath $MountPath had been umounted!"
        else
                echo "MountPath $MountPath had not been mounted!"
        fi
else
        echo "MountPath $MountPath does not existed!"
fi

5. 卸載範例

/PATH/TO/umount_UFS_snapshot.sh /SNAPSHOT/Mount/Path/Weekly0

6. UFS 做 snapshot 速度還蠻慢的. (相較於 ZFS 的快速完成 snapshot)

參考資料
FreeBSD UFS/ZFS Snapshot Management Environment

Saturday, February 18, 2012

在 FreeBSD 使用 portmaster 更新已安裝的 ports - 程式碼修改

程式碼:

#!/usr/local/bin/bash

PKGLISTPATH="/PATH/TO/STORE/PKGLIST"

if [ -n "$1" ]; then
        keyword="$1"
        shift
else
        echo "Usage: $0 keyword"
        echo "Example: $0 lib"
        exit 1
fi

echo $keyword

if [ -f $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt ]
then
        echo "Had $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt"
else
        echo "Generating $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt"
        portversion -l '<' > $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt
fi

for i in $(cat $PKGLISTPATH/pkgversion_`date +"%Y%m%d"`.txt | grep $keyword | awk '{ print $1 }'); do portmaster -D $i; done

備份系統設定且紀錄差異

為了玩玩在 shell script 使用函式功能, 寫了一隻簡單的程式碼紀錄系統設定的差異. 不過增加項目還是需要修改程式碼. 暫時先能用再來看如何修改.
程式碼:

#!/usr/local/bin/bash

BasePath="/PATH/TO/BACKUP/DIRECTORY"
PATH=$PATH

BackupConfig(){
if [ -z $1 ]
# Checks if any params.
then
 echo "No parameters passed to function."
 return 0
else
 ProgName=$1
fi

if [ $2 ]
then
 BackupFolder=$2
fi

case $ProgName in
postfix) CMD="postconf -n"
  ;;
sysctl)  CMD="sysctl -a"
  ;;
pkglist) CMD="ls /var/db/pkg"
  ;;
*)  echo "Do not support this program $ProgName"
                ;;
esac

mkdir -p $BasePath/$BackupFolder
rm -f $BasePath/$BackupFolder/${ProgName}_old.txt

echo "Check if had OLD data : "
if [ -f $BasePath/$BackupFolder/${ProgName}_now.txt ]; then
 echo "   Had OLD data! Rename it to ${ProgName}_old.txt"
 mv $BasePath/$BackupFolder/${ProgName}_now.txt $BasePath/$BackupFolder/${ProgName}_old.txt
else
 echo "   Do not have OLD data."
fi

echo "Backup configuration of ${ProgName} now."
$CMD > $BasePath/$BackupFolder/${ProgName}_now.txt

echo "Generate diff log : "
if [ -f $BasePath/$BackupFolder/${ProgName}_old.txt ]; then
 echo "   Generate diff log between ${ProgName}_new.txt and ${ProgName}_old.txt."
 diff $BasePath/$BackupFolder/${ProgName}_now.txt $BasePath/$BackupFolder/${ProgName}_old.txt > $BasePath/$BackupFolder/diff-`date +"%Y%m%d"`.txt
 find  $BasePath/$BackupFolder/ -type f -size 0 | xargs rm -f
else
 echo "   Do not have OLD data."
fi

}


### Backup postfix setting
BackupConfig postfix postfix_config_backup

### Backup sysctl setting
BackupConfig sysctl sysctl_config_backup

### Backup /var/db/pkg
BackupConfig pkglist installed-pkglist

Monday, February 13, 2012

自動化抓衛星雲圖(全球及氣象局) 更新版

1.抓取全球衛星雲圖(Hemisphere)程式碼:


#!/usr/local/bin/bash

### http://www.opentopia.com/sunlightmaprect.html
### http://www.opentopia.com/sunlightmaphemi.html
### http://static.die.net/earth/mercator/1600.jpg
### http://static.die.net/earth/hemisphere/1600.jpg

DATESTR=`date "+%Y%m%d%H%M"`
LASTMONTH=`date -v -1m +"%Y%m"`
THISMONTH=`date "+%Y%m"`
LASTYEAR=`date -v -1y +"%Y"`
THISYEAR=`date "+%Y"`
WEBROOTDIR="/PATH/to/Web/Root/Directory"
BASEDIR="$WEBROOTDIR/world_sunlight_map"
TEMPDIR="/tmp"
WGETLOC="/usr/local/bin/wget"
USERAGENT='--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"'
CONVERTLOC="/usr/local/bin/convert"

#下載全球衛星雲圖
###$WGETLOC -q -O $WEBROOTDIR/world_sunlight_map_ori.jpg http://www.opentopia.com/images/data/sunlight/world_sunlight_map_rectangular.jpg
###$WGETLOC -q -O $WEBROOTDIR/world_sunlight_map_hemisphere_ori.jpg http://www.opentopia.com/images/data/sunlight/world_sunlight_map_hemisphere.jpg
$WGETLOC -q ${USERAGENT} -O $WEBROOTDIR/world_sunlight_map_ori.jpg http://static.die.net/earth/mercator/1600.jpg
$WGETLOC -q ${USERAGENT} -O $WEBROOTDIR/world_sunlight_map_hemisphere_ori.jpg http://static.die.net/earth/hemisphere/1600.jpg
/bin/rm -f $WEBROOTDIR/world_sunlight_map_rectangular.jpg
/bin/rm -f $WEBROOTDIR/world_sunlight_map_hemisphere.jpg
/bin/mv $WEBROOTDIR/world_sunlight_map_ori.jpg $WEBROOTDIR/world_sunlight_map_rectangular.jpg
/bin/mv $WEBROOTDIR/world_sunlight_map_hemisphere_ori.jpg $WEBROOTDIR/world_sunlight_map_hemisphere.jpg
/bin/cp $WEBROOTDIR/world_sunlight_map_rectangular.jpg ${BASEDIR}/world_sunlight_map_rectangular_${DATESTR}.jpg
/bin/cp $WEBROOTDIR/world_sunlight_map_hemisphere.jpg ${BASEDIR}/world_sunlight_map_hemisphere_${DATESTR}.jpg
/bin/cp $WEBROOTDIR/world_sunlight_map_rectangular.jpg ${TEMPDIR}/original.jpg

#切割、合併圖片, 讓東亞在地圖中間
$CONVERTLOC ${TEMPDIR}/original.jpg -crop 600x887+0+0 ${TEMPDIR}/left.jpg
$CONVERTLOC ${TEMPDIR}/original.jpg -crop 1600x887+600+0 ${TEMPDIR}/right.jpg
$CONVERTLOC ${TEMPDIR}/right.jpg ${TEMPDIR}/left.jpg +append ${BASEDIR}/TW/world_sunlight_map.jpg

/bin/cp ${BASEDIR}/TW/world_sunlight_map.jpg ${BASEDIR}/TW/world_sunlight_map/world_sunlight_map_${DATESTR}.jpg
/bin/rm -f ${TEMPDIR}/original.jpg ${TEMPDIR}/left.jpg ${TEMPDIR}/right.jpg

2. 抓取中央氣象局 V7 東亞雲圖程式碼:


#!/usr/local/bin/bash

PATH=$PATH

### http://www.cwb.gov.tw/V7/observe/satellite/Sat_H_EA.htm?type=1#

WEBROOTDIR="/PATH/to/Web/Root/Directory"
WGETLOC="/usr/local/bin/wget"

###中央氣象局衛星雲圖
CWBWEBBASEURL="http://www.cwb.gov.tw/V7/observe/satellite/Data"
CWBBASEDIR="$WEBROOTDIR/CWB"
CWBDATE=`date +"%Y-%m-%d-%H-"`
CWBDATE1="${CWBDATE}00"
CWBDATE2="${CWBDATE}30"
NOWMINSEC=`date +"%H%M"`
USERAGENT='--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"'

###中央氣象局衛星雲圖
# MTSAT 紅外線雲圖      HS1P
# MTSAT 紅外線 色調強化 HS1Q
# MTSAT 紅色線 黑白     HS1O
# MTSAT 可見光          HSAO

for MapType in HS1P HS1Q HS1O HSAO
do
        if [ -f ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg ]; then
                echo "${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg was existed."
        else
                if [ ${MapType} == 'HSAO' ]; then
                        if [ $NOWMINSEC -gt 0600 ]&&[ $NOWMINSEC -le 1930 ]; then
                                $WGETLOC -q ${USERAGENT} -O ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg $CWBWEBBASEURL/${MapType}/${MapType}-${CWBDATE1}.jpg
                        fi
                else
                        $WGETLOC -q ${USERAGENT} -O ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg $CWBWEBBASEURL/${MapType}/${MapType}-${CWBDATE1}.jpg
                fi
        fi

        if [ -f ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE2}.jpg ]; then
                echo "${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE2}.jpg was existed."
        else
                if [ ${MapType} == 'HSAO' ]; then
                        if [ $NOWMINSEC -gt 0600 ]&&[ $NOWMINSEC -le 1930 ]; then
                                $WGETLOC -q ${USERAGENT} -O ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg $CWBWEBBASEURL/${MapType}/${MapType}-${CWBDATE1}.jpg
                        fi
                else
                        $WGETLOC -q ${USERAGENT} -O ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE2}.jpg $CWBWEBBASEURL/${MapType}/${MapType}-${CWBDATE2}.jpg
                fi
        fi

        sleep 2
done

# Remove empty file
find  ${CWBBASEDIR}/ -type f -size 0 | xargs rm -f

舊版程式碼:

#!/usr/local/bin/bash

### http://www.opentopia.com/sunlightmaprect.html
### http://www.opentopia.com/sunlightmaphemi.html
### http://www.cwb.gov.tw/V7/observe/satellite/Sat_H_EA.htm?type=1#

DATESTR=`date "+%Y%m%d%H%M"`
LASTMONTH=`date -v -1m +"%Y%m"`
THISMONTH=`date "+%Y%m"`
LASTYEAR=`date -v -1y +"%Y"`
THISYEAR=`date "+%Y"`
WEBROOTDIR="/PATH/to/Web/Root/Directory"
BASEDIR="$WEBROOTDIR/world_sunlight_map"
TEMPDIR="/tmp"
WGETLOC="/usr/local/bin/wget"
CONVERTLOC="/usr/local/bin/convert"

###中央氣象局衛星雲圖
CWBWEBBASEURL="http://www.cwb.gov.tw/V7/observe/satellite/Data"
CWBBASEDIR="$WEBROOTDIR/CWB"
CWBDATE=`date +"%Y-%m-%d-%H-"`
CWBDATE1="${CWBDATE}00"
CWBDATE2="${CWBDATE}30"
USERAGENT='--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"'

#下載全球衛星雲圖
$WGETLOC -q -O $WEBROOTDIR/world_sunlight_map_ori.jpg http://www.opentopia.com/images/data/sunlight/world_sunlight_map_rectangular.jpg
$WGETLOC -q -O $WEBROOTDIR/world_sunlight_map_hemisphere_ori.jpg http://www.opentopia.com/images/data/sunlight/world_sunlight_map_hemisphere.jpg
/bin/rm -f $WEBROOTDIR/world_sunlight_map_rectangular.jpg
/bin/rm -f $WEBROOTDIR/world_sunlight_map_hemisphere.jpg
/bin/mv $WEBROOTDIR/world_sunlight_map_ori.jpg $WEBROOTDIR/world_sunlight_map_rectangular.jpg
/bin/mv $WEBROOTDIR/world_sunlight_map_hemisphere_ori.jpg $WEBROOTDIR/world_sunlight_map_hemisphere.jpg
/bin/cp $WEBROOTDIR/world_sunlight_map_rectangular.jpg ${BASEDIR}/world_sunlight_map_rectangular_${DATESTR}.jpg
/bin/cp $WEBROOTDIR/world_sunlight_map_hemisphere.jpg ${BASEDIR}/world_sunlight_map_hemisphere_${DATESTR}.jpg
/bin/cp $WEBROOTDIR/world_sunlight_map_rectangular.jpg ${TEMPDIR}/original.jpg

#切割、合併圖片, 讓東亞在地圖中間
$CONVERTLOC ${TEMPDIR}/original.jpg -crop 600x887+0+0 ${TEMPDIR}/left.jpg
$CONVERTLOC ${TEMPDIR}/original.jpg -crop 1600x887+600+0 ${TEMPDIR}/right.jpg
$CONVERTLOC ${TEMPDIR}/right.jpg ${TEMPDIR}/left.jpg +append ${BASEDIR}/TW/world_sunlight_map.jpg

/bin/cp ${BASEDIR}/TW/world_sunlight_map.jpg ${BASEDIR}/TW/world_sunlight_map/world_sunlight_map_${DATESTR}.jpg
/bin/rm -f ${TEMPDIR}/original.jpg ${TEMPDIR}/left.jpg ${TEMPDIR}/right.jpg

sleep 5

###中央氣象局衛星雲圖
# MTSAT 紅外線雲圖      HS1P
# MTSAT 紅外線 色調強化 HS1Q
# MTSAT 紅色線 黑白     HS1O
# MTSAT 可見光          HSAO

for MapType in HS1P HS1Q HS1O HSAO
do
 if [ -f ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg ]; then
  echo "${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg was existed."
 else
  $WGETLOC -q ${USERAGENT} -O ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE1}.jpg $CWBWEBBASEURL/${MapType}/${MapType}-${CWBDATE1}.jpg
 fi

 if [ -f ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE2}.jpg ]; then
  echo "${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE2}.jpg was existed."
 else
  $WGETLOC -q ${USERAGENT} -O ${CWBBASEDIR}/${MapType}/${MapType}-${CWBDATE2}.jpg $CWBWEBBASEURL/${MapType}/${MapType}-${CWBDATE2}.jpg
 fi

 sleep 5
done

# Remove empty file
find  ${CWBBASEDIR}/ -type f -size 0 | xargs rm -f

Monday, January 30, 2012

Shell script 搭配 ipfw table 阻擋入侵 sshd 攻擊 authlog_banip.sh 修正版

1. 目前已將之前只有阻擋 30 天改為先阻擋 30 天, 若是 30 ~ 90 天前仍有多筆紀錄就再加鎖一段時間直到符合 Good Boy 條件.

2. 程式碼如下:
For SSHD


#!/usr/local/bin/bash

LOGFILE="/PATH/TO/auth.log"
PROCFOLDER="/PATH/TO/auth_sshd_banip"
TOTALBANLOGFILE="/PATH/TO/auth_sshd_banip.log"
TOTALBANLOGFILESORTED="/PATH/TO/auth_sshd_banip_sorted.log"
TOTALBANLOGFILEFROMPROCFOLDERSORTED="/PATH/TO/auth_sshd_banip_from_procfolder_sorted.log"
TOTALRESCUELIST="/PATH/TO/auth_sshd_rescue_list.log"
CRONJOBDATE="crontjobdate.log"
TODAYLOGFM=`date +"%b %e"`
DefaultCheckPeriod="30"
MVFilePeriod="90"

TODAY=`date +"%Y%m%d"`
OLDDAY=`date -v-${DefaultCheckPeriod}d +"%Y%m%d"`
OLDYEAR=`date -v-${DefaultCheckPeriod}d +"%Y"`

fwcmd="/sbin/ipfw"

echo "Prepare folder and file"
mkdir -p $PROCFOLDER
touch $TOTALBANLOGFILE
touch $TOTALRESCUELIST
touch $PROCFOLDER/$CRONJOBDATE
touch $TOTALBANLOGFILEFROMPROCFOLDERSORTED

SSHDErrorCounter1=`cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Invalid user" | awk '{ print $10 }' | wc -l`
SSHDErrorCounter2=`cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Did not receive identification string from" | awk '{ print $12 }' | wc -l`
SSHDErrorCounter3=`cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "not allowed because none of user" | awk '{ print $9 }' | wc -l`

if [ $SSHDErrorCounter1 -ne 0 ]||[ $SSHDErrorCounter2 -ne 0 ]||[ $SSHDErrorCounter3 -ne 0 ]||[ -f $PROCFOLDER/sshdpre-$TODAY.log ]; then
        echo "Write pre-process file"
        touch $PROCFOLDER/sshdpre-$TODAY.log
        cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Invalid user" | awk '{ print $10 }' | sort -n | sort -u >> $PROCFOLDER/sshdpre-$TODAY.log
        cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Did not receive identification string from" | awk '{ print $12 }' | sort -n | sort -u >> $PROCFOLDER/sshdpre-$TODAY.log
        cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "not allowed because none of user" | awk '{ print $9 }' | sort -n | sort -u >> $PROCFOLDER/sshdpre-$TODAY.log
        cp -a $PROCFOLDER/sshdpre-$TODAY.log /tmp
        cat /tmp/sshdpre-$TODAY.log | sort -n | sort -u > $PROCFOLDER/sshdpre-$TODAY.log
        rm -f /tmp/sshdpre-$TODAY.log

        echo "Collect new ban IP"
        for ip in $(awk '{ print }' $PROCFOLDER/sshdpre-$TODAY.log)
        do
                if [ `grep $ip $PROCFOLDER/sshd-*.log | wc -l` -gt 0 ]; then
                        echo $ip "Old bad boy"
                else
                        echo $ip
                        
                        touch $PROCFOLDER/sshd-$TODAY.log
                        echo $ip >> $PROCFOLDER/sshd-$TODAY.log
                        echo $ip >> $TOTALBANLOGFILE
                fi
        done

        echo "Ban bad boy"
        if [ -f $PROCFOLDER/sshd-$TODAY.log ]; then
                for banip in $(awk '{ print }' $PROCFOLDER/sshd-$TODAY.log)
                do
                        #Ban IP
                        echo "Check $banip"

                        if [ `ipfw table 3 list | grep $banip | wc -l` -gt 0 ]; then
                                echo "Had ban bad boy($banip)."
                        else
                                echo "Ban bad boy($banip) now."
                                ${fwcmd} table 3 add $banip/32
                                ${fwcmd} table 4 add $banip/32
                        fi
                done
        else
                echo "No bad boy"
        fi

        echo "Rescue good boy"
        if [ -f $PROCFOLDER/sshd-$OLDDAY.log ]; then
                for rescueip in $(awk '{ print }' $PROCFOLDER/sshd-$OLDDAY.log)
                do
                        #Rescue IP
                        echo "Check $rescueip"
                        
                        BadBoyDCPCounter=`find $PROCFOLDER -d 1 -ctime +${DefaultCheckPeriod} -type f -name 'sshd*.log' -exec grep $rescueip {} \; | wc -l`
                        BadBoyMVFCounter=`find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'sshd*.log' -exec grep $rescueip {} \; | wc -l`

                        if [ $BadBoyDCPCounter -lt 1 ]; then
                                if [ $BadBoyMVFCounter -le 2 ]; then
                                        echo "Rescue $rescueip"
                                        ${fwcmd} table 3 delete $rescueip/32
                                        ${fwcmd} table 4 delete $rescueip/32
                                
                                        grep -v $rescueip $TOTALBANLOGFILE > /tmp/stillbanip-$TODAY
                                        cp -f /tmp/stillbanip-$TODAY $TOTALBANLOGFILE
                                        rm -f /tmp/stillbanip-$TODAY
                                        echo $rescueip >> $TOTALRESCUELIST
                                        echo "Rescued"
                                else
                                        echo "Not bad boy in ${DefaultCheckPeriod} days, but had bad boy record in ${MVFilePeriod} days."
                                fi
                        else
                                echo "Still bad boy in ${DefaultCheckPeriod} days"
                        fi
                done
                
                #Move to old folder
                echo "Move old file to storage folder"
                mkdir -p $PROCFOLDER/$OLDYEAR
                mv $PROCFOLDER/sshd*-$OLDDAY.log $PROCFOLDER/$OLDYEAR
        else
                echo "No candidate file"
        fi

        cat $TOTALBANLOGFILE | sort -n | sort -u > $TOTALBANLOGFILESORTED
        
else
        echo "No bad boy"
fi

### Move old files to backup folder
find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'sshd*.log' -exec mv {} $PROCFOLDER/$OLDYEAR \;
cat $PROCFOLDER/sshd-*.log | sort -n | sort -u > $TOTALBANLOGFILEFROMPROCFOLDERSORTED

### Additional rescure goodboy
for rescueip in `diff $TOTALBANLOGFILEFROMPROCFOLDERSORTED $TOTALBANLOGFILESORTED | grep -v a | awk '{ print $2 }' | sort -r -n`
do
        echo "Additional rescure goodboy from diff $TOTALBANLOGFILEFROMPROCFOLDERSORTED and $TOTALBANLOGFILESORTED"     
        echo "Rescue $rescueip"
        ${fwcmd} table 3 delete $rescueip/32
        ${fwcmd} table 4 delete $rescueip/32

        grep -v $rescueip $TOTALBANLOGFILE > /tmp/stillbanip-$TODAY
        cp -f /tmp/stillbanip-$TODAY $TOTALBANLOGFILE
        rm -f /tmp/stillbanip-$TODAY
        echo $rescueip >> $TOTALRESCUELIST
        echo "Rescued"
done

cat $TOTALBANLOGFILE | sort -n | sort -u > $TOTALBANLOGFILESORTED

echo $TODAY > $PROCFOLDER/$CRONJOBDATE

### For backup and administration
/PATH/TO/BACKUP_SCRIPT

For MAIL


#!/usr/local/bin/bash

PATH=$PATH

LOGFILE="/PATH/TO/maillog"
DOVECOTERRORLOGFILE="/PATH/TO/dovecot/error.log"
DOVECOTINFOLOGFILE="/PATH/TO/dovecot/info.log"
PROCFOLDER="/PATH/TO/mail_banip"
DAILYLOG="mail_banip"`date +"%Y%m%d"`".log"
TMPFILE="/tmp/mailban-"`date +"%s"`
TODAYLOGFM=`date +"%b %e"`
fwcmd="/sbin/ipfw"

DefaultCheckPeriod="30"
MVFilePeriod="30"

OLDDAILYLOG="mail_banip"`date -v-${DefaultCheckPeriod}d +"%Y%m%d"`".log"
OLDYEAR=`date -v-${MVFilePeriod}d +"%Y"`

mkdir -p $PROCFOLDER

MAILErrorString1="NOQUEUE: reject"
MAILErrorCounter1=`cat $LOGFILE | grep "$TODAYLOGFM" | grep postfix | grep "$MAILErrorString1" | wc -l`
MAILErrorString2="auth-worker(default): Error: pam"
MAILErrorCounter2=`cat $DOVECOTERRORLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString2" | awk 'BEGIN { FS = ": " } ; { print $3 }' | awk 'BEGIN { FS = "," } ; { print $2 }' | sort -n | sort -u | sed -Ee 's/)//' | wc -l`
MAILErrorString3="pop3-login: Info: Aborted login (auth failed"
MAILErrorCounter3=`cat $DOVECOTINFOLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString3" | awk 'BEGIN { FS = ", " } ; { print $4 }' | awk 'BEGIN { FS = "=" } ; { print $2 }' | sort -n | sort -u | wc -l`

if [ $MAILErrorCounter1 -ne 0 ]||[ $MAILErrorCounter2 -ne 0 ]||[ $MAILErrorCounter3 -ne 0 ]||[ -f $PROCFOLDER/$DAILYLOG ]; then
        if [ ! -f $PROCFOLDER/$DAILYLOG ]; then
                touch $PROCFOLDER/$DAILYLOG
        fi

        echo "Collect Bad boy"
        touch $TMPFILE
        cat $LOGFILE | grep "$TODAYLOGFM" | grep postfix | grep "NOQUEUE: reject" | awk 'BEGIN { FS = "[" } ; { print $3 }' | awk 'BEGIN { FS = "]" } ; { print $1 }' | sort -n | sort -u >> $PROCFOLDER/$DAILYLOG
        cat $DOVECOTERRORLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString2" | awk 'BEGIN { FS = ": " } ; { print $3 }' | awk 'BEGIN { FS = "," } ; { print $2 }' | sort -n | sort -u | sed -Ee 's/)//' >> $PROCFOLDER/$DAILYLOG
        cat $DOVECOTINFOLOGFILE | grep "$TODAYLOGFM" | grep "$MAILErrorString3" | awk 'BEGIN { FS = ", " } ; { print $4 }' | awk 'BEGIN { FS = "=" } ; { print $2 }' | sort -n | sort -u >> $PROCFOLDER/$DAILYLOG
        cat $PROCFOLDER/$DAILYLOG | sort -n | sort -u | grep -v "192.168.58" | grep -v "192.168.68" | grep -v "192.168.8" | grep -v "60.248.16.64" | grep -v "60.248.16.65" | grep -v "60.248.16.66" > $TMPFILE
        cp -f $TMPFILE $PROCFOLDER/$DAILYLOG

        echo "Ban Bad Boy"
        for banip in $(awk '{ print }' $PROCFOLDER/$DAILYLOG)
        do
                if [ `ipfw table 3 list | grep $banip | wc -l` -gt 0 ]; then
                        echo "Had ban bad boy($banip)."
                else
                        echo "Ban bad boy($banip) now."
                        ${fwcmd} table 3 add $banip/32
                        ${fwcmd} table 4 add $banip/32
                fi
        done
        
        rm -f $TMPFILE
fi

if [ -f $PROCFOLDER/$OLDDAILYLOG ]; then
        for rescueip in $(awk '{ print }' $PROCFOLDER/$OLDDAILYLOG)
        do
                echo "Rescue $rescueip"
                ${fwcmd} table 3 delete $rescueip/32
                ${fwcmd} table 4 delete $rescueip/32
        done
else
        echo "No candidate file"
fi

#Move to old folder
echo "Move old file to storage folder"
mkdir -p $PROCFOLDER/$OLDYEAR
find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'mail_banip*.log' -exec mv {} $PROCFOLDER/$OLDYEAR \;

For HTTPD


#!/usr/local/bin/bash

PATH=$PATH

LOGFILE="/PATH/TO/httpd-access.log"
PROCFOLDER="/PATH/TO/httpd_banip"
DAILYLOG="httpd_banip"`date +"%Y%m%d"`".log"
TMPFILE="/tmp/httpdban-"`date +"%s"`
TODAYLOGFM=`date +"%d/%b/%Y"`
fwcmd="/sbin/ipfw"

DefaultCheckPeriod="30"
MVFilePeriod="30"

OLDDAILYLOG="httpd_banip"`date -v-${DefaultCheckPeriod}d +"%Y%m%d"`".log"
OLDYEAR=`date -v-${MVFilePeriod}d +"%Y"`

mkdir -p $PROCFOLDER

ErrorString1="xmlrpc.php"
ErrorCounter1=`cat $LOGFILE | grep "$TODAYLOGFM" | grep "$ErrorString1" | grep 404 | awk 'BEGIN { FS = " " } ; { print $1 }' | grep -v "192.168.58" | grep -v "192.168.68" | grep -v "192.168.8" | grep -v "60.248.16.64" | grep -v "60.248.16.65" | grep -v "60.248.16.66" | wc -l`

if [ $ErrorCounter1 -ne 0 ]||[ -f $PROCFOLDER/$DAILYLOG ]; then
        if [ ! -f $PROCFOLDER/$DAILYLOG ]; then
                touch $PROCFOLDER/$DAILYLOG
        fi

        echo "Collect Bad boy"
        touch $TMPFILE
        cat $LOGFILE | grep "$TODAYLOGFM" | grep "$ErrorString1" | grep 404 | awk 'BEGIN { FS = " " } ; { print $1 }' | sort -n | sort -u >> $PROCFOLDER/$DAILYLOG
        cat $PROCFOLDER/$DAILYLOG | sort -n | sort -u | grep -v "192.168.58" | grep -v "192.168.68" | grep -v "192.168.8" | grep -v "60.248.16.64" | grep -v "60.248.16.65" | grep -v "60.248.16.66" > $TMPFILE
        cp -f $TMPFILE $PROCFOLDER/$DAILYLOG
        
        echo "Ban Bad Boy"
        for banip in $(awk '{ print }' $PROCFOLDER/$DAILYLOG)
        do
                if [ `ipfw table 3 list | grep $banip | wc -l` -gt 0 ]; then
                        echo "Had ban bad boy($banip)."
                else
                        echo "Ban bad boy($banip) now."
                        ${fwcmd} table 3 add $banip/32
                        ${fwcmd} table 4 add $banip/32
                fi
        done
        
        rm -f $TMPFILE
fi

if [ -f $PROCFOLDER/$OLDDAILYLOG ]; then
        for rescueip in $(awk '{ print }' $PROCFOLDER/$OLDDAILYLOG)
        do
                echo "Rescue $rescueip"
                ${fwcmd} table 3 delete $rescueip/32
                ${fwcmd} table 4 delete $rescueip/32
        done
else
        echo "No candidate file"
fi

#Move to old folder
echo "Move old file to storage folder"
mkdir -p $PROCFOLDER/$OLDYEAR
find $PROCFOLDER -d 1 -ctime +${MVFilePeriod} -type f -name 'httpd_banip*.log' -exec mv {} $PROCFOLDER/$OLDYEAR \;

3. 以上程式碼為自由軟體, 請自行取用. 強烈建議服用前先看清楚再服用, 避免發生意外.

Friday, September 30, 2011

如何掛載 FreeBSD 光碟影像檔

Script 如下 :

mail(xxx)# more mount_FreeBSDDVD.sh
#!/bin/sh

mkdir -p /FreeBSD-RELEASEDVD
mdconfig -a -t vnode -f /PATH/TO/FreeBSD-IMAGE.iso -u 1
mount -t cd9660 /dev/md1 /FreeBSD-RELEASEDVD

mail(xxx)# more umount_FreeBSDDVD.sh
#!/bin/sh

umount /FreeBSD-RELEASEDVD
mdconfig -d -u 1

Monday, September 05, 2011

在 FreeBSD 使用 portmaster 更新已安裝的 ports

在 bash 下完整跑完所有已安裝的 ports

for i in $(ls /var/db/pkg/); do portmaster -D $i; done
或
for i in `ls /var/db/pkg/`; do portmaster -D $i; done

在 bash 下針對需要更新版本的 ports

pkg_version -v > /PATH/TO/pkg_version_`date +"%Y%m%d"`.txt
grep "needs updating" /PATH/TO/pkg_version_`date +"%Y%m%d"`.txt > /PATH/TO/pkg_version_needs-updating_`date +"%Y%m%d"`.txt
for i in $(awk '{ print $1 }' /PATH/TO/pkg_version_needs-updating_`date +"%Y%m%d"`.txt); do portmaster -D $i; done

較簡易版

portversion -l '<' > /root/pkg_list/pkgversion_`date +"%Y%m%d"`.txt
for i in $(awk '{ print $1 }' /root/pkg_list/pkgversion_`date +"%Y%m%d"`.txt); do portmaster -D $i; done

也可以將 portmaster -D 改用 pkg_upgrade 取代.

Sunday, May 22, 2011

產生 OPIE key 的 script

程式 opiekey.sh

#!/usr/local/bin/bash

CHKPARA=0

if [ -n "$1" ]; then
        CCounter="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        Seed="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        Pass="$1"
        shift
else
        CHKPARA=1
fi

if [ -n "$1" ]; then
        Mailto="$1"
        shift
else
        Mailto="ACCOUNT@DOMAIN.NAME"
fi

if [ $CHKPARA -eq 1 ]; then
        echo "Usage: $0 CCounter Seed Pass [Mailto]"
        echo "Example: $0 496 ma7133 hiugwhig"
        exit 1
else
        echo "Generate password now."
fi

echo "CCounter Seed "$CCounter" "$Seed
echo "Pass "$Pass
echo "Mailto "$Mailto

echo $Pass | opiekey -n 1 $CCounter $Seed | mail -s OTPKey $Mailto

自動化抓衛星雲圖(全球及氣象局)

1. 程式 get_world_sunlight_map.sh

#!/bin/sh

### http://www.opentopia.com/sunlightmaprect.html
### http://www.opentopia.com/sunlightmaphemi.html

DATESTR=`date "+%Y%m%d%H%M"`
BASEDIR="/usr/local/www/data/world_sunlight_map"
TEMPDIR="/tmp"

###中央氣象局衛星雲圖
CWBBASEDIR="/usr/local/www/data/CWBV6"
CWBDATE=`date +"%Y-%m-%d-%H-%M"`
USERAGENT='--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"'

#下載全球衛星雲圖
/usr/local/bin/wget -q -O /usr/local/www/data/world_sunlight_map_ori.jpg http://www.opentopia.com/images/data/sunlight/world_sunlight_map_rectangular.jpg
/bin/rm -f /usr/local/www/data/world_sunlight_map_rectangular.jpg
/bin/mv /usr/local/www/data/world_sunlight_map_ori.jpg /usr/local/www/data/world_sunlight_map_rectangular.jpg
/bin/cp /usr/local/www/data/world_sunlight_map_rectangular.jpg ${BASEDIR}/world_sunlight_map_rectangular_${DATESTR}.jpg
/bin/cp /usr/local/www/data/world_sunlight_map_rectangular.jpg ${TEMPDIR}/original.jpg

#切割、合併圖片, 讓東亞在地圖中間
/usr/local/bin/convert ${TEMPDIR}/original.jpg -crop 600x887+0+0 ${TEMPDIR}/left.jpg
/usr/local/bin/convert ${TEMPDIR}/original.jpg -crop 1600x887+600+0 ${TEMPDIR}/right.jpg
/usr/local/bin/convert ${TEMPDIR}/right.jpg ${TEMPDIR}/left.jpg +append ${BASEDIR}/TW/world_sunlight_map.jpg

/bin/cp ${BASEDIR}/TW/world_sunlight_map.jpg ${BASEDIR}/TW/world_sunlight_map/world_sunlight_map_${DATESTR}.jpg
/bin/rm -f ${TEMPDIR}/original.jpg ${TEMPDIR}/left.jpg ${TEMPDIR}/right.jp

sleep 10

###中央氣象局衛星雲圖
# MTSAT 紅外線雲圖
/usr/local/bin/wget -q ${USERAGENT} -O ${CWBBASEDIR}/HS1P/HS1P-${CWBDATE}.jpg http://www.cwb.gov.tw/V6/observe/satellite/Data/HS1P/HS1P-${CWBDATE}.jpg

sleep 10

# MTSAT 紅外線 色調強化
/usr/local/bin/wget -q ${USERAGENT} -O ${CWBBASEDIR}/HS1Q/HS1Q-${CWBDATE}.jpg http://www.cwb.gov.tw/V6/observe/satellite/Data/HS1Q/HS1Q-${CWBDATE}.jpg

sleep 10

# MTSAT 紅色線 黑白
/usr/local/bin/wget -q ${USERAGENT} -O ${CWBBASEDIR}/HS1O/HS1O-${CWBDATE}.jpg http://www.cwb.gov.tw/V6/observe/satellite/Data/HS1O/HS1O-${CWBDATE}.jpg

sleep 10

# MTSAT 可見光
/usr/local/bin/wget -q ${USERAGENT} -O ${CWBBASEDIR}/HSAO/HSAO-${CWBDATE}.jpg http://www.cwb.gov.tw/V6/observe/satellite/Data/HSAO/HSAO-${CWBDATE}.jpg

# Remove empty file
find  ${CWBBASEDIR}/ -type f -size 0 | xargs rm -f

2. 在 crontab 加上(不用太常連線抓檔)

30      */2     *       *       *       root    /PATH/TO/get_world_sunlight_map.sh

3. 想要在 Windows XP 換桌面可以改用 Desktop Earth.

Shell script 從 Apache log 使用關鍵字蒐集來源 IP

程式碼如下

#!/usr/local/bin/bash

if [ -n "$1" ]; then
        keyword="$1"
        shift
else
        echo "Usage: $0 keyword"
        echo "Example: $0 Googlebot"
        exit 1
fi


touch /var/log/webot.$keyword
touch /var/log/$keyword.txt

#Backup old data
mv /usr/local/www/data/$keyword-*.txt /usr/local/www/data/webbots
cp -a /var/log/$keyword.txt /usr/local/www/data/$keyword-`date +"%Y%m%d"`.txt

#Process data
cat /var/log/$keyword.txt > /var/log/webot.$keyword
grep $keyword /var/log/httpd-access.log | awk '{ print $1 }' | sort -n | sort -u >> /var/log/webot.$keyword
cat /var/log/webot.$keyword | sort -n | sort -u > /var/log/$keyword.txt

#Remove temp file
rm -f /var/log/webot.$keyword

Shell script 搭配 ipfw table 阻擋入侵 sshd 攻擊

自己用來偷懶的 script. 自動化使用 ipfw table 阻擋 IP 也會在定義時間後判斷是否有繼續受到攻擊, 再決定是否將該 IP 解除鎖定.

1. ipfw table 語法

${fwcmd} add 06000 set 5 deny log tcp from table\(3\) to any dst-port PORT1,PORT2 in via ${INTIF}
${fwcmd} add 06000 set 5 deny log tcp from table\(4\) to any dst-port PORT11,PORT12 in via ${INTIF}

2. authlog_banip_init.sh (只需要執行一次)

#!/usr/local/bin/bash

LOGFILE="/var/log/auth.log"
PROCFOLDER="/var/log/auth_sshd_banip"
TOTALBANLOGFILE="/var/log/auth_sshd_banip.log"
TODAYLOGFM=`date +%b %d`

TODAY=`date +"%Y%m%d"`
OLDDAY=`date -v-30d +"%Y%m%d"`
OLDYEAR=`date -v-30d +"%Y"`

fwcmd="/sbin/ipfw"

echo "Prepare folder and file"
mkdir -p $PROCFOLDER
touch $TOTALBANLOGFILE

if [ `cat $LOGFILE | grep sshd | grep "Invalid user" | awk '{ print $10 }' | sort -n | sort -u | wc -l` -ne 0 ]; then
 echo "Write pre-process file"
 cat $LOGFILE | grep sshd | grep "Invalid user" | awk '{ print $10 }' | sort -n | sort -u > $PROCFOLDER/sshdpre-$TODAY.log

 echo "Collect new ban IP"
 for ip in $(awk '{ print }' $PROCFOLDER/sshdpre-$TODAY.log)
 do
  if [ `grep $ip $PROCFOLDER/sshd-*.log | wc -l` -gt 0 ]; then
   echo $ip "Old bad boy"
  else
   echo $ip
   
   touch $PROCFOLDER/sshd-$TODAY.log
   echo $ip >> $PROCFOLDER/sshd-$TODAY.log
   echo $ip >> $TOTALBANLOGFILE
  fi
 done

 echo "Ban bad boy"
 if [ -f $PROCFOLDER/sshd-$TODAY.log ]; then
  for banip in $(awk '{ print }' $PROCFOLDER/sshd-$TODAY.log)
  do
   #Ban IP
   echo $banip
   
   ${fwcmd} table 3 add $banip/32
   ${fwcmd} table 4 add $banip/32
  done
 else
  echo "No bad boy"
 fi

 echo "Rescue good boy"
 if [ -f $PROCFOLDER/sshd-$OLDDAY.log ]; then
  for rescueip in $(awk '{ print }' $PROCFOLDER/sshd-$OLDDAY.log)
  do
   #Rescue IP
   echo $rescueip
   
   if [ `grep $rescueip $PROCFOLDER/sshd-*.log | wc -l` -eq 1 ]; then
    ${fwcmd} table 3 delete $rescueip/32
    ${fwcmd} table 4 delete $rescueip/32
    
    grep -v $rescueip $TOTALBANLOGFILE > /tmp/stillbanip-$TODAY
    cp /tmp/stillbanip-$TODAY $TOTALBANLOGFILE
    echo "Rescued"
   else
    echo "Still bad boy"
   fi
  done
  
  #Move to old folder
  echo "Move old file to storage folder"
  mkdir -p $PROCFOLDER/$OLDYEAR
  mv $PROCFOLDER/sshd*-$OLDDAY.log $PROCFOLDER/$OLDYEAR
 else
  echo "No candidate file"
 fi
 
else
 echo "No bad boy"
fi

3. authlog_banip.sh

#!/usr/local/bin/bash

LOGFILE="/var/log/auth.log"
PROCFOLDER="/var/log/auth_sshd_banip"
TOTALBANLOGFILE="/var/log/auth_sshd_banip.log"
TODAYLOGFM=`date +%b %d`

TODAY=`date +"%Y%m%d"`
OLDDAY=`date -v-30d +"%Y%m%d"`
OLDYEAR=`date -v-30d +"%Y"`

fwcmd="/sbin/ipfw"

echo "Prepare folder and file"
mkdir -p $PROCFOLDER
touch $TOTALBANLOGFILE

if [ `cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Invalid user" | awk '{ print $10 }' | sort -n | sort -u | wc -l` -ne 0 ]; then
 echo "Write pre-process file"
 cat $LOGFILE | grep "$TODAYLOGFM" | grep sshd | grep "Invalid user" | awk '{ print $10 }' | sort -n | sort -u > $PROCFOLDER/sshdpre-$TODAY.log

 echo "Collect new ban IP"
 for ip in $(awk '{ print }' $PROCFOLDER/sshdpre-$TODAY.log)
 do
  if [ `grep $ip $PROCFOLDER/sshd-*.log | wc -l` -gt 0 ]; then
   echo $ip "Old bad boy"
  else
   echo $ip
   
   touch $PROCFOLDER/sshd-$TODAY.log
   echo $ip >> $PROCFOLDER/sshd-$TODAY.log
   echo $ip >> $TOTALBANLOGFILE
  fi
 done

 echo "Ban bad boy"
 if [ -f $PROCFOLDER/sshd-$TODAY.log ]; then
  for banip in $(awk '{ print }' $PROCFOLDER/sshd-$TODAY.log)
  do
   #Ban IP
   echo $banip
   
   ${fwcmd} table 3 add $banip/32
   ${fwcmd} table 4 add $banip/32
  done
 else
  echo "No bad boy"
 fi

 echo "Rescue good boy"
 if [ -f $PROCFOLDER/sshd-$OLDDAY.log ]; then
  for rescueip in $(awk '{ print }' $PROCFOLDER/sshd-$OLDDAY.log)
  do
   #Rescue IP
   echo $rescueip
   
   if [ `grep $rescueip $PROCFOLDER/sshd-*.log | wc -l` -eq 1 ]; then
    ${fwcmd} table 3 delete $rescueip/32
    ${fwcmd} table 4 delete $rescueip/32
    
    grep -v $rescueip $TOTALBANLOGFILE > /tmp/stillbanip-$TODAY
    cp /tmp/stillbanip-$TODAY $TOTALBANLOGFILE
    echo "Rescued"
   else
    echo "Still bad boy"
   fi
  done
  
  #Move to old folder
  echo "Move old file to storage folder"
  mkdir -p $PROCFOLDER/$OLDYEAR
  mv $PROCFOLDER/sshd*-$OLDDAY.log $PROCFOLDER/$OLDYEAR
 else
  echo "No candidate file"
 fi
 
else
 echo "No bad boy"
fi

4. 在 crontab 加上

59      23      *       *       *       root    /PATH/TO/authlog_banip.sh

5. 開機時執行 authlog_banip.sh
5.1. 在 /etc/rc.local 加上
### For Firewall
/PATH/to/authlog_banip_boot.sh

5.2. authlog_banip_boot.sh

#!/usr/local/bin/bash

TOTALBANLOGFILE="/var/log/auth_sshd_banip.log"
fwcmd="/sbin/ipfw"

if [ -f $TOTALBANLOGFILE ]; then
 for banip in $(awk '{ print }' $TOTALBANLOGFILE)
 do
  #Ban IP
  echo $banip
  
  ${fwcmd} table 3 add $banip/32
  ${fwcmd} table 4 add $banip/32
 done
else
 echo "No bad boy"
fi

Jengyic's 正義部落