Attachment 5971 Details for Bug 34984

[patch] New aproach: split uncompressed date into 4k blocks and check them seperatly, so we can keep the progress functionality of the original code

ucc-image-toolkit.diff (text/plain), 1.30 KB, created by Drees Dormann on 2014-06-26 12:07 CEST

(hide)

Description:

Filename:

MIME Type:

Creator: Drees Dormann

Created: 2014-06-26 12:07 CEST

Size: 1.30 KB

patch

obsolete

>Index: src/images.py
>===================================================================
>--- src/images.py	(Revision 51251)
>+++ src/images.py	(Arbeitskopie)
>@@ -158,6 +158,7 @@
> 	try:
> 		with contextlib.nested(open(infile, 'rb'), open(outfile, 'wb')) as (fin, fout):
> 			counter = 0
>+			BLOCK_SIZE = 4096  # block size is dependent on file system block size for sparse file
> 			while True:
> 				compressed_data = fin.read(DEFAULT_CHUNK_SIZE)
> 				if not compressed_data:
>@@ -165,10 +166,20 @@
> 					break
> 
> 				uncompressed_data = decompressor.decompress(compressed_data)
>-				fout.write(uncompressed_data)
>+
>+				# don't write blocks that consist only of 0-bytes to keep sparse file format
>+				for i in xrange(0, len(uncompressed_data), BLOCK_SIZE):
>+					if len(uncompressed_data[i:i+BLOCK_SIZE]) == BLOCK_SIZE:
>+						if uncompressed_data[i:i+BLOCK_SIZE].count('\x00') == BLOCK_SIZE:
>+							fout.seek(BLOCK_SIZE, os.SEEK_CUR)
>+						else:
>+							fout.write(uncompressed_data[i:i+BLOCK_SIZE])
>+					# decompressed data can not be always a multiple of BLOCK_SIZE
>+					# so write rest data to avoid data loss
>+					else:
>+						fout.write(uncompressed_data[i:])
>+
> 				progress(fin.tell(), total_size)
>-				del compressed_data
>-				del uncompressed_data
> 
> 				# check free size on disk every N rounds
> 				if counter % 50 == 0:

Actions: View | Diff

Attachments on bug 34984: 5970 | 5971 | 5985