suse linux下修复文件系统,Suse 11 下一次文件系统修复的案例 .

操作系统：Suse Linux 11

文件系统：ext3

错误现象

X日，接到告警，检查文件系统/dev/sda1发现写入报只读，检查IP存储有告警，随即umount /img，但卸载后无法正常挂载

fdisk -l显示IO错误，重启应用服务器后依然无法正常挂载，显示IO错误，

检查IP存储有告警信息，待存储厂商解决存储问题后，重启应用服务器仍然无法正常挂载文件系统，

由于mount命令执行后长时间无响应，但观察/var/log/messages仍然显示系统在进行block的扫描：

Nov 2 06:04:53 linux11 kernel: [128293.578670] Buffer I/O error on device sda1, logical block 483584660

Nov 2 06:04:53 linux11 kernel: [128293.578672] lost page write due to I/O error on sda1

Nov 2 06:05:01 linux11 /usr/sbin/cron[15283]: (root) CMD ( /opt/hp/hp-health/bin/check-for-restart-requests)

Nov 2 06:05:53 linux11 kernel: [128353.584893] sd 9:0:0:0: [sda] Unhandled sense code

Nov 2 06:05:53 linux11 kernel: [128353.584898] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Nov 2 06:05:53 linux11 kernel: [128353.584901] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

Nov 2 06:05:53 linux11 kernel: [128353.584905] sd 9:0:0:0: [sda] Add. Sense: Medium not present

Nov 2 06:05:53 linux11 kernel: [128353.584910] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 e6 97 59 5f 00 00 08 00Nov 2 06:05:53 linux11 kernel: [128353.584916] end_request: I/O error, dev sda, sector 3868678495Nov 2 06:05:53 linux11 kernel: [128353.584920] Buffer I/O error on device sda1, logical block 483584804

Nov 2 06:05:53 linux11 kernel: [128353.584922] lost page write due to I/O error on sda1

Nov 2 06:05:53 linux11 kernel: [128353.599875] sd 9:0:0:0: [sda] Unhandled sense code

Nov 2 06:05:53 linux11 kernel: [128353.599878] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Nov 2 06:05:53 linux11 kernel: [128353.599880] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

Nov 2 06:05:53 linux11 kernel: [128353.599883] sd 9:0:0:0: [sda] Add. Sense: Medium not present

Nov 2 06:05:53 linux11 kernel: [128353.599886] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 e6 97 5f bf 00 00 08 00

Nov 2 06:05:53 linux11 kernel: [128353.599890] end_request: I/O error, dev sda, sector 3868680127

Nov 2 06:05:53 linux11 kernel: [128353.599893] Buffer I/O error on device sda1, logical block 483585008

Nov 2 06:05:53 linux11 kernel: [128353.599895] lost page write due to I/O error on sda1

Nov 2 06:05:53 linux11 kernel: [128353.600872] sd 9:0:0:0: [sda] Unhandled sense code

Nov 2 06:05:53 linux11 kernel: [128353.600875] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Nov 2 06:05:53 linux11 kernel: [128353.600877] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

Nov 2 06:05:53 linux11 kernel: [128353.600879] sd 9:0:0:0: [sda] Add. Sense: Medium not present

Nov 2 06:05:53 linux11 kernel: [128353.600882] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 e6 97 62 47 00 00 08 00

Nov 2 06:05:53 linux11 kernel: [128353.600887] end_request: I/O error, dev sda, sector 3868680775

红色部分显示系统仍在工作中，等待20小时候，工程师建议继续等待，20小时后，mount命令运行结束

linux11:~ #mount /dev/sda1 /mnt/mount: wrong fs type, bad option, bad superblock on /dev/sda1,

missing codepage or helper program, or other error

In some cases useful info is found in syslog - try

dmesg | tail or so

linux11:~ #dmesg|tail -50[138764.297170] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138764.297172] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138764.297175] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138764.297178] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 1f f5 b7 00 00 10 00

[138764.297182] end_request: I/O error, dev sda, sector 4062180791

[138764.312193] sd 9:0:0:0: [sda] Unhandled sense code

[138764.312197] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138764.312199] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138764.312202] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138764.312204] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 20 37 9f 00 00 08 00

[138764.312209] end_request: I/O error, dev sda, sector 4062197663

[138764.312224] sd 9:0:0:0: [sda] Unhandled sense code

[138764.312226] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138764.312228] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138764.312230] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138764.312233] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 20 38 b7 00 00 08 00

[138764.312237] end_request: I/O error, dev sda, sector 4062197943

[138764.312242] sd 9:0:0:0: [sda] Unhandled sense code

[138764.312243] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138764.312245] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138764.312247] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138764.312250] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 20 7f 87 00 00 08 00

[138764.312254] end_request: I/O error, dev sda, sector 4062216071

[138824.286688] sd 9:0:0:0: [sda] Unhandled sense code

[138824.286692] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138824.286696] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138824.286699] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138824.286704] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 20 f2 bf 00 00 08 00

[138824.286710] end_request: I/O error, dev sda, sector 4062245567

[138824.286714] __ratelimit: 8 callbacks suppressed

[138824.286718] Buffer I/O error on device sda1, logical block 507780688

[138824.286719] lost page write due to I/O error on sda1

[138824.324706] sd 9:0:0:0: [sda] Unhandled sense code

[138824.324709] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138824.324711] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138824.324714] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138824.324717] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 20 fa 1f 00 00 08 00

[138824.324722] end_request: I/O error, dev sda, sector 4062247455

[138824.324726] Buffer I/O error on device sda1, logical block 507780924

[138824.324727] lost page write due to I/O error on sda1

[138824.324741] sd 9:0:0:0: [sda] Unhandled sense code

[138824.324742] sd 9:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

[138824.324744] sd 9:0:0:0: [sda] Sense Key : Medium Error [current]

[138824.324747] sd 9:0:0:0: [sda] Add. Sense: Medium not present

[138824.324749] sd 9:0:0:0: [sda] CDB: Write(10): 2a 00 f2 2e a1 17 00 00 08 00

[138824.324754] end_request: I/O error, dev sda, sector 4063142167

[138824.324756] Buffer I/O error on device sda1, logical block 507892763

[138824.324758] lost page write due to I/O error on sda1

[138824.324773] JBD: recovery failed

[138824.324774] EXT3-fs: error loading journal.

修复方案

工程师初步判定为Superblock损坏，开始进行制定修复方案：

1.通过dd将原/dev/sda1分区的文件备份到其他文件分区，原分区大小2T，IP存储重新划分了略大于2T的空间，挂到应用服务器上，进行数据备份

2.数据备份后通过fsck.ext3进行修复

一、数据备份

创建新的分区/dev/sdb1

linux11:/var/log #fdisk /dev/sdb

The number of cylinders for this disk is set to 267075.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-267075, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-267075, default 267075):

Using default value 267075

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

linux11:/var/log #

linux11:/var/log #

linux11:/var/log #

linux11:/var/log # fdisk -l

Disk /dev/cciss/c0d0: 300.0 GB, 299966445568 bytes

255 heads, 63 sectors/track, 36468 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x000bf615

Device Boot Start End Blocks Id System

/dev/cciss/c0d0p1 * 1 38 305203+ 83 Linux

/dev/cciss/c0d0p2 39 4215 33551752+ 82 Linux swap / Solaris

/dev/cciss/c0d0p3 4216 36468 259072222+ 83 Linux

Disk /dev/sda: 2097.2 GB, 2097152000000 bytes

255 heads, 63 sectors/track, 254964 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x000a13a0

Device Boot Start End Blocks Id System

/dev/sda1 1 254964 2047998298+ 83 Linux

Disk /dev/sdb: 2196.8 GB, 2196766720000 bytes

255 heads, 63 sectors/track, 267075 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x24828d3f

Device Boot Start End Blocks Id System

/dev/sdb1 1 267075 2145279906 83 Linux

注意，这里尝试使用了mkfs格式化文件分区，由于文件系统2T，格式化时间相当长，最终取消了这一操作，注意kill操作也不能很快的结束，只有等待，随即重新划分了存储空间，进行分区，但不进行格式化

开始数据备份

dd if=/dev/sda1 of=/dev/sdb1 bs=8M

最开始的时候未指定bs的大小，默认只有512字节，经过约30小时的等待后，测速发现只有1M/s，后中断该过程，改为bs=8M

应用服务器未安装stat包，补充测速的方法：

>strace.log

time strace -o strace.log -p 11929

运行一段时间后ctrl+c终止

统计write出现的次数

grep -c write starace.log

echo "次数*8/time得到的时间" |bc

即为估算的每秒复制的速度。

30个小时后备份结束

249999+1 records in

249999+1 records out

2097150257664 bytes (2.1 TB) copied, 130468 s, 16.1 MB/s

二、用备份数据进行恢复

由于原应用服务器还使用临时空间在承担业务，因此通过IP存储将分区挂载到其他操作系统相同的机器进行修复，首先确定超级块superblock的起始位置

linux11:~ #dumpe2fs /dev/sda1

dumpe2fs 1.41.9 (22-Aug-2009)

Filesystem volume name:

Last mounted on:

Filesystem UUID: 34689bab-428f-4e84-b3b8-22351dfcbe9a

Filesystem magic number: 0xEF53

Filesystem revision #: 1 (dynamic)

Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file

Filesystem flags: signed_directory_hash

Default mount options: (none)

Filesystem state: clean

Errors behavior: Continue

Filesystem OS type: Linux

Inode count: 128000000

Block count: 511999574

Reserved block count: 25599978

Free blocks: 483185304

Free inodes: 126463697

First block: 0

Block size: 4096

Fragment size: 4096

Reserved GDT blocks: 901

Blocks per group: 32768

Fragments per group: 32768

Inodes per group: 8192

Inode blocks per group: 512

Filesystem created: Wed May 11 16:02:51 2011

Last mount time: Thu Aug 2 17:26:01 2012

Last write time: Thu Aug 2 17:26:01 2012

Mount count: 8

Maximum mount count: -1

Last checked: Wed May 11 16:02:51 2011

Check interval: 0 ()

Reserved blocks uid: 0 (user root)

Reserved blocks gid: 0 (group root)

First inode: 11

Inode size: 256

Required extra isize: 28

Desired extra isize: 28

Journal inode: 8

Default directory hash: half_md4

Directory Hash Seed: 7c5d0a45-f12f-4ce5-8e8a-cb1029acbf2d

Journal backup: inode blocks

Journal size: 128MGroup 0: (Blocks 0-32767)Primary superblock at 0, Group descriptors at 1-123

Reserved GDT blocks at 124-1024

Block bitmap at 1025 (+1025), Inode bitmap at 1026 (+1026)

Inode table at 1027-1538 (+1027)

1255 free blocks, 7617 free inodes, 14 directories

Free blocks: 4442, 31514-32767

Free inodes: 576-8192Group 1: (Blocks 32768-65535)Backup superblock at 32768, Group descriptors at 32769-32891

Reserved GDT blocks at 32892-33792

Block bitmap at 33793 (+1025), Inode bitmap at 33794 (+1026)

Inode table at 33795-34306 (+1027)

7263 free blocks, 7415 free inodes, 16 directories

Free blocks: 34340-36863, 37296, 49851-52059, 52083-54611

Free inodes: 8970-16384Group 2: (Blocks 65536-98303)Block bitmap at 65536 (+0), Inode bitmap at 65537 (+1)

Inode table at 65538-66049 (+2)

4 free blocks, 7403 free inodes, 21 directories

Free blocks: 72068-72071

Free inodes: 17174-24576

Group 3: (Blocks 98304-131071)

...有很多个superblock,见红色字体部分，以下省略...

操作系统将超级块备份到了多个位置，本次选择用32768处的进行修复

linux01:~ #fsck.ext3 -y -b 32768 /dev/sda1e2fsck 1.41.9 (22-Aug-2009)

Superblock needs_recovery flag is clear, but journal has data.

Recovery flag not set in backup superblock, so running journal anyway.

/dev/sda1: recovering journal

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

Block bitmap differences: -(22281864--22282239) -(22282760--22282957) -(22282967--22283536) -(22283552--22284122) -(22284138--22284287) -(22284582--22285018) -(100555611--100556799) -(100580404--100581255)

Fix? yes

Free blocks count wrong for group #0 (31223, counted=1255).

Fix? yes

Free blocks count wrong for group #1 (31229, counted=7263).

Fix? yes

Free blocks count wrong for group #2 (32254, counted=4).

Fix? yes

Free blocks count wrong for group #3 (31229, counted=0).

Fix? yes

...省略...

Free inodes count wrong for group #15622 (8192, counted=7167).

Fix? yes

Directories count wrong for group #15622 (0, counted=34).

Fix? yes

Free inodes count wrong for group #15623 (8192, counted=6821).

Fix? yes

Directories count wrong for group #15623 (0, counted=52).

Fix? yes

Free inodes count wrong for group #15624 (8192, counted=7247).

Fix? yes

Directories count wrong for group #15624 (0, counted=21).

Fix? yes

Free inodes count wrong (127999989, counted=120656202).

Fix? yes

/dev/sda1: ***** FILE SYSTEM WAS MODIFIED *****

/dev/sda1: 7343798/128000000 files (2.8% non-contiguous), 259583311/511999574 blocks

修复成功，重新挂载文件系统，能够正常加载，文件和目录能够正常访问