Skill Sprint: Practical Data Compression and Decompression

Posted by on in Programming

Delphi Skill Sprint Replay


Video not found or Youtube service not available
C++ Skill Sprint Replay

Video not found or Youtube service not available

Background on Data Compression

Most media compression (images, videos, audio, etc.) uses lossy compression, which is specialized compression that removes part of the media in a way that only degrades the media slightly. For data compression, a lossless compression is used so that the data is preserved through the compression process.

Most lossless data compression is based on two papers by Abraham Lempel and Jacob Ziv published in 1977 and 1978. These papers are referred to as LZ77 & LZ78. Derived compression algorithms are often referred to as LZW, LZSS, LZMA, etc.

The .ZIP file format was developed by Phil Katz to store multiple files with different compression options. The most popular compression option was deflateLempel–Ziv–Welch (LZW) is the algorithm behind deflate. The Unix compress utility and the GIF image file format were based on same deflate implementation. Later it was discovered this implementation was encumbered by patents.

Gzip was developed as an open and alternative implementation of deflate that avoids the patents on compress and GIF. It is used as the basis for the new PNG image file format. 

ZLib is a library that wraps gzip's deflate. It supports 3 modes: raw deflate without headers; Gzip with headers and PNG image file format.

For more information:

HTTP Compression Options

  • On the request, Accept-Encoding specifies expected compression(s).
  • In the response, Content-Encoding specifies the actual compression.
  • Most common encodings: 
    • gzip & deflate (gzip is much more common)
  • Other encodings (rare): 
    • compres, exi, identity, pack200-gzip, xz, bzip2, lzma, xpress
    • brotli is a new encoding by Google that is 20% more effective for HTML
  • Accept-Encoding on the request may be ignored. 
  • Stack Overflow always compresses with gzip

Compression Library Options

  • Preinstalled 
    • System.Zlib – supports gzip (with proper header) and deflate 
    • Indy’s TIdCompressorZLib (also based on Zlib)
      • HTTP GZip, HTTP Deflate, FTPToIO and FTPDeflate
  • Available via GetIt
    • Abbrevia (all platforms)
      • Huge variety of compressions, including Zip files
      • Has nice test suite with usage examples
      • Local folder: $(BDSCatalogRepository)\Abbrevia-10.0\tests
        • BDSCatalogRepository = C:\Users\[user]\Documents\Embarcadero\Studio\17.0\CatalogRepository\
    • JEDI JCL – JclCompression (Windows only)
      • More compressions, including 7zip files

Code Samples

 



About
Gold User, Rank: 15, Points: 256
This latest season of Developer Skill Sprints focuses on programming tips and techniques most requested by attendees of the past 72 skill sprints we’ve already presented. This quarter we’ll take your

Comments

  • CESAR G. XMAPAS
    CESAR G. XMAPAS Tuesday, 9 May 2017

    The problem is that i can't get data on the client side. (RESTRespLog.contentlength = 0 , RESTRespLog.ContentEncoding='gzip')

  • CESAR G. XMAPAS
    CESAR G. XMAPAS Tuesday, 9 May 2017

    Hello. This is what i've been looking for so long, but here's the question:
    I have a webbroker server with the apis for my client, everything go ok, but i'm trying to compress my jsonarray data from the server to be used on my client (using the same components as in your example) but i can't get it work fine. So i leave my code from the server side for your help please:

    procedure TWModMain.LoginGet(Request: TWebRequest; Response: TWebResponse);
    var
    a: TJSONArray; //array de json
    o: TJSONObject; //objects del json array
    MemStr, gZIPStream: TMemoryStream; // for gzip
    begin
    //Response.Content := ' {Test OK} ' ;

    // Load query parameters.
    qry.SQL.clear;
    if (Request.QueryFields.values['_IdUser'] '') then
    begin
    qry.SQL.Text := 'SELECT a.* FROM usuarios a ' +
    ' WHERE a.id_usuario = "' + Request.QueryFields.values['_IdUser'] + '"';
    end
    else qry.SQL.Text := 'SELECT a.* FROM usuarios a ';


    // Put results in a JSON object...
    FDConnection1.Connected:=True;
    qry.Active := true;
    if qry.Active then
    begin
    if qry.RecordCount>0 then
    begin
    a := TJSONArray.Create;
    try
    qry.First;
    while (not qry.Eof) do
    begin
    o := TJSONObject.Create;
    o.AddPair('_IdUser',TJSONString.Create( qry.FieldByName('id_usuario').AsString ));
    o.AddPair('_Password',qry.FieldByName('Password').AsString );
    o.AddPair('_Name',qry.FieldByName('FirstName').AsString + ' ' + qry.FieldByName('LastName').AsString );
    a.AddElement(o);
    qry.Next;
    end;
    finally
    //****** parameter from the request client side
    // compress the data and return gzip file
    if Request.QueryFields.values['_zip'] = '' then
    begin
    //Init
    MemStr:= TMemoryStream.Create;
    gZIPStream:=TMemoryStream.Create;
    try
    MemStr.Seek(0, soFromBeginning);
    //convert jsonarray to Tstream
    WriteStreamStr(MemStr, Utf8ToAnsi(a.ToString));
    //make it gzip from (http://stackoverflow.com/questions/40047200/delphi-how-to-set-twebresponse-to-contain-gzip-and-read-it-in-android-app)
    doGZIP(MemStr,gZIPStream);
    //prepare responsestream and set content encoding and type
    Response.Content := '';
    Response.ContentStream:=gZIPStream;
    Response.ContentEncoding := 'gzip';
    Response.ContentType := 'application/json';
    finally
    //clear...
    MemStr.Free;
    gZIPStream.free;
    end;
    end else
    begin
    Response.ContentType := 'application/json';
    Response.Content := Utf8ToAnsi(a.ToString); //a.ToString; //
    end;
    //close data and release vars
    a.DisposeOf;
    qry.Active := False;
    FDConnection1.Connected:=False;
    end;
    end;
    end;
    end;

  • Please login first in order for you to submit comments
  • Page :
  • 1

Check out more tips and tricks in this development video: