Delphi

Better Performance with REST Compression

Many Delphi applications, esp. mobile iOS or Android apps, are using REST, to retrieve data from a backend. Often TRESTClient and TRESTRequest are used to get access to an external REST api. There several Blogs and CodeRage videos (including from me), that demonstrate how to do this. Even the Delphi online documentation has a fairly simple tutorial on this topic:

Delphi Songsterr REST Tutorial

Most of these sample have on thing in common: They are slower as possible!

Most REST APIs are hosted on more or less modern WEb server, be it IIS, Apache, Nginx or whatever. All those support HTTP(S) compression. The actual api implementation usually doesn’t know about that.

Following the example from Delphi’s documentation, after executing the following request from a Webbrowser and investigating the transferred data in the browser’s Web console, you will probably think like “Excellent, that chatty XML has been compressed down to just about 10%”. And indeed this may result in an important speedup for your app.

https://www.songsterr.com/a/ra/songs.xml?pattern=Marley

In Firefox’ Web Console you can easily identify the compression – obviously the XML data is shrunk down to less than 10%:

Komprimierte Daten

If you now do the same test with Delphi’s RESTDebugger (which internally uses TRESTClient), then the problem gets apparent immediately:

Clearly, data is transferred uncompressed here – slowly that is. The reason is, that an HTTP-Sever usually doesn’t compress data “just so”. The browser/client has to ask for it. Common Web browsers automatically ask the server to compress data. Delphi’s TRESTClient does not do this automatically.

Solution:

procedure TForm1.Button1Click(Sender: TObject);
var
  LValue:TJSONValue;
begin
  RESTRequest1.AcceptEncoding := 'gzip, deflate, br';
  RESTRequest1.Execute;
  LValue:=RESTResponse1.JSONValue;
  MemoContent.Text:= LValue.ToString;
end;

This asks the Web server to compress data, preferably using “gzip”. You can try that in Delphi’s RESTDebugger by adding a “Header Parameter” like that:

and – surprise – the server sends back compressed data:

Setting Accept-Encoding via TRESTRequest.AcceptEncoding or via Header-Parameter, does not matter. Important though, check that ” Do not encode” checkbox in RESTDebugger. Or set “DoNotEncode” in your source code (without that, the spaces and commas will be URL encoded) :

  //Either like this
  LRequest.AcceptEncoding := 'gzip, deflate, br';
  //OR like this
  LParam := LRequest.Params.AddHeader('Accept-Encoding', 'br, gzip, deflate');
  LParam.Options := [poDoNotEncode];
  //-->> same result!

 

Signing Windows Delphi Applications

FireMonkey Apps for iOS and Android are signed automatically by Delphi during the build process – this is an obligation esp. for iOS apps and thus has been added as a feature directly into the IDE.

For Windows applications though (32 or 64, VCL or FMX) there is no such “Sign” option in Delphi. This most certainly leads to the fact that many Delphi Apps are getting deployed unsigned.

English translation in progress – stay tuned …

Bei Windows Anwendungen (egal ob 32 oder 64 Bit) ist dagegen keine Signatur-Option in Delphi zu finden. Dies dürfte dazu führen, dass viele Delphi Apps unsigniert verteilt werden. Dies ist jedoch spätestens seit Windows 10 nicht mehr zu empfehlen, denn der mit Windows 10 eingeführte “SmartScreen-Filter” blockiert standardmäßig “nicht vertrauenswürdige” Anwendungen. Eine unsignierte Anwendung (EXE) ist per se “nicht vertrauenswürdig”, da ihre Herkunft nicht verifiziert werden kann.

Hin und wieder findet man dann den wenig schlauen “Tip”, wie man diese SmartScreen Prüfung abschalten kann. Einzig richtige Vorgehensweise ist es dagegen, als Entwickler nur signierte Anwendungen heraus zu geben (egal ob öffentlich oder inhouse)! Als Anwender sollte man Lieferanten auf die Füße treten, die weiterhin unsignierte Anwendungen verkaufen.

Signatur-Zertifikate

Um eine EXE zu signieren, ist zunächst ist ein valides Zertifikat erforderlich, welches das sog. Code Sigining unterstützt. Solche Zertifikate werden von Anbietern wie Comodo, Thawte, Symantec und anderen herausgegeben. Die Anbieter führen dabei je nach gewählter Preisklasse eine mehr oder minder intensive Indentitätsprüfung des Beantragenden durch – schließlich soll das Zertifikat ja eindeutig die Firma oder Person, die eine Anwendung liefert, identifizieren können. Die Kosten liegen dabei jährlich im unteren bis mittleren dreistelligen Bereich.

Brauchbare kostenlose Zertifikate konnte ich nicht finden (so scheut z.B. LetsEncrypt den Aufwand, den eine Identitätsprüfung mit sich bringt), die günstigste Variante ist ein Comodo-Zertifikat per CodeSignCert zu bestellen. Dort ist man mit $75 für ein Jahr dabei.

Signieren in Delphi

Zunächst ist das entsprechende Windows 10 SDK zu installieren, welches die sog. “signtool.exe” von Microsoft enthält. Die 32-Bit Version wird sich danach typischerweise in einem Pfad ähnlich diesem finden : C:\Program Files\Windows Kits\10\bin\x86\signtool.exe

Nun ist zu überlegen, wie mit dem Zertifikat umgegangen werden soll. Es gibt zwei Möglichkeiten:

  1. Installation im Windows-Zertifikatspeicher und damit Nutzung des Windows-eigenen Schutzmechanismusses.
  2. Speichern des Zertifikats als PFX-Datei und Hinterlegen des Passworts im Build-Prozess der Anwendung.

Letztlich muss bedacht werden, dass dieses Zertifikat wie ein Personalausweis zu verstehen ist. Man sollte es also nicht einfach “irgendwo rumliegen” lassen, da sonnst jedermann diese Identität für seine eigenen Zwecke nutzen könnte. Besonder das Einchecken in ein Sourcecontrol-System scheint hier wenig sinnvoll.

Ich nutze in der Regel den Windows-Zertifikatspeicher, dort habe ich dann eine zentrale Übersicht meiner Zertifikate und kann so auch schnell sehen, wann ein Zertifikat abläuft.

Grundsätzlich kann man nun eine kompilierte Delphi-Anwendung von Hand signieren, indem man folgenden Befehl auf der Kommandozeile eingibt:

"C:\Program Files\Windows Kits\10\bin\x86\signtool.exe" sign /n "developer experts" /tr http://timestamp.comodoca.com /du https://www.developer-experts.net C:\Projekte\SignTest\Win32\Release\SignTest.exe

Die SignTest.exe wird dadurch mit einem Zertifikat signiert, welches im “Subject” (dem Namen) “Developer Experts” enhält. Wenn sich nur ein einziges Code Signing Zertifikat im Windows Zertifikatspeicher befindet, dann kann man die Option /n auch weg lassen – es schadet aber auch nicht.

Die weiteren Optionen:
/tr – hier wird ein sog. Timestam-Server angegeben. Verschiedene Zertifikatslieferanten (die sog. CAs) betreiben diese, ich nutze hier den Server von Comodo – es muss jedoch nicht zwingend der Server des eigenen Zertifikatslieferanten sein. Durch diese Option wird (kurz formuliert) dem Timestamp-Server ein Hash meiner Exe übermittelt, der Server stellt dann eine Signatur zu diesem Hash aus, die mit einem, auf dem Server generierten, Zeitstempel versehen wird. Dies wird dann der eigentlichen Signatur hinzugefügt. Das Verfahren soll verhindern, dass der Lieferant einer EXE nicht “schummeln” kann, also nicht vor- oder rückdatieren kann.
Achtung: Verschiedene ältere Blogeinträge zu diesem Thema nutzen meinst nur die Option /t – dies führt jedoch – ohne weitere Angabe eines Algorithmus – zur Nutzung des veralteten SHA1 Verfahrens. Die Option /tr dagegen arbeitet nach RFC 3161 und handelt mit dem Server mindestens SHA256 aus.

/du – fügt die angegebene URL als “erweiterte Beschreibung” der Signatur hinzu. Dies ist insbesondere für eine Hersteller-URL geeignet

Um diesen Signiervorgang nicht jedes mal manuell ausführen zu müssen, kann man das in die Post-Build-Option des jeweiligen Delphi Projekts einbinden. Ich nutze hierfür die Release-Konfiguration, weil nur diese zum Kunden geht.

Zu beachten ist, dass man hier praktischerweise “($OUTPUTPATH)” als Signaturziel angibt, damit der Pfad nicht hartkodiert ist – und man das ganze per Copy and Paste in andere Projekte einfügen kann 😉

Ausserdem ist das Pre-Build-Ereignis “del ($OUTPUTPATH)” wichtig, weil sonst mitunter das Signieren fehlschlägt.

Mit jedem Kompilieren” wird nun in der Release-Konfiguration die EXE signiert – die Debug-Konfiguration bleibt unverändert. Gerade bei sehr großen Projekten kann eine Debug-Exe schon mal schnell mehrere hundert MB groß werden, was beim Signieren dann auch ein wenig Zeit benötigt.

Setting the correct Xcode build number for Delphi FMX Apps

If your Xcode version updates (manually or by Apple’s updating mechanism) then make sure to re-import the iOS/macOS SDK from within Delphi.

To build and deploy iOS/macOS apps with Delphi, you need Xcode for the final steps, even though the actual binary is compiled by Delphi. Apple frequently delivers minor updates to Xcode. Current version as of the writing of this article is 8.3.2. This version number of the Xcode build used for preparing apps is apparently checked by Apple, when uploading an IPA file to App Store/iTunes Connect. They don’t do any spooky things, they just check the DTXcodeBuild key in your app’s info.plist file:

 

This info.plist file is generated by Delphi, and contains various essential settings, such as version number, device requirements etc. Many of these settings can directly be configured, by modifying the values in Delphi – Project – Options – Version Information

DTXCodeBuild is filled in by Delphi automatically though. When you import iOS/macOS SDK, then PAServer obviously issues this command:

/usr/bin/xcodebuild -version -sdk

That returns the available SDK versions and the Xcode build number:

iPhoneOS10.3.sdk - iOS 10.3 (iphoneos10.3)
SDKVersion: 10.3
Xcode 8.3.2
Build version 8E2002

This build version number is apparently stored and used to fill the DTXcodeBuild key’s value.

To bring this number in Delphi in sync with the actual Xcode version, after Xcode was updated (or switched with xcode-select), you have to delete the SDK from Delphi and re-import it using Delphi – Tools – Options – SDK Manager.

This is especially important if you imported the 10.3 SDK while having Xcode 8.3.0 installed. In that case your Delphi iOS apps would be tagged with being built with exactly that version – no matter if you downgraded to Xcode 8.2.1 or applied my “Package Application fix” – which is required due to a tool chain change that Apple imposed with Xcode 8.3 and up. Any iOS app tagged with being built with Xcode 8.3.0 will be refused by Apple, as that version has been deprecated.

Delphi and Xcode 8.3.x deployment solution

Since Apple updated Xcode to version 8.3.x, iOS IPA deployment is broken with Delphi up to 10.2 (Tokyo). There is an official workaround, which basically instructs to download Xcode 8.2 and use that for now:

http://docwiki.embarcadero.com/PlatformStatus/en/Main_Page#iOS_10

Unfortunately though, Apple started sending out notifications, that it won’t accept any builds created with Xcode versions older than 8.3.2 anymore. This is what I’ve received from Apple a few days ago:

“Dear developer,
We have discovered one or more issues with your recent delivery for “AppEvents”. Your delivery was successful, but you may wish to correct the following issues in your next delivery:
Deprecated Xcode Build – Due to resolved app archives issues, we will be deprecating Xcode 8.3 on May 10, 2017, at which time app archives built with 8.3 will no longer be accepted by the App Store. Download Xcode 8.3.2 or newer, rebuild your app and resubmit.
After you’ve corrected the issues, you can use Xcode or Application Loader to upload a new binary to iTunes Connect.
Regards,
The App Store team”

The problem with Delphi and Xcode 8.3 is basically a tool, that Apple decided to remove from its tool chain, but which Delphi still depends on. So the solution is to copy that missing tool “PackageApp” from 8.2 to 8.3. Below are the steps that made my Delphi 10.2 (Tokyo) iOS deployment fully working again:

  1. Install current Xcode from Appstore into your applications folder – currently 8.3.2
  2. Download Xcode 8.2.1 from
    https://developer.apple.com/download/more/
  3. Unzip Xcode_8.2.1.xip to a separate folder (e.g. /tmp/Xcode)
  4. Make a backup copy of your current Xcode version:
    cp -R /Applications/Xcode.app /tmp/Xcode.8.3.app (might take a minute or two)
  5. Copy the following file from 8.2.1 to your current Xcode app:
    sudo cp /tmp/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/PackageApplication /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/

Do not forget to add Export/Encryption exemption to your Delphi iOS apps

Apple requires you to specify if your apps are using any form of encryption. If you do not use encryption, you have to confirm that in your app settings on iTunes Connect – this is not new, Apple started asking for that long time ago. Important to notice is though that since a while Apple requires your app to have a special entry in it’s info.plist file – even if do not use any encryption. Without that your app will fail to verify for Testflight and Appstore deliveries.

 

It’s easy to add that entry in Delphi. Go to Project – Options – Version information. The list/grid is actually editable, just go to the last row and press cursor down. Then add “ITSAppUsesNonExemptEncryption – false” as shown in the screenshot

FireDAC BatchMove and Calculated Fields

If you need to “pump” records from one data source to an other, then TFDBatchmove is for you.

Using FireDAC’s BatchMove component allows you to easily map the fields of one query to an other, regardless if they belong to the same database/connection or two totally different databases. You can actually even copy records from text files to “real” databases and vice versa.

Just build a BatchMove chain like this:

SourceData and DataTarget are regular TFDQueries, Reader and Writer are TFDBatchmoveDatasetReader/Writer components, that just connect the queries with the TFDBatchmove component in the middle.

The actual magic is done by adding field mappings, from source to target:

In this case source and target fieldnames are mostly the same, which wouldn’t even require adding mappings. There are a couple of calculated fields in the source query too. They are used to format/modify the data, so that it would match the format requirements in the destiantion query.

These fields are calculated in QOPCQuery.OnCalcFields like this:

procedure TDMMain.QOPCSourceDataCalcFields(DataSet: TDataSet);
begin
  if QOPCSourceDataoSchnittgruppe.IsNull then
    DataSet.FieldByName('FANr').AsString := ''
  else
    DataSet.FieldByName('FANr').AsString := GenFANr(QOPCSourceDataoSchnittgruppe.AsInteger); 

The calculated fields work fine, but when calling BatchMove.execute it will raise an exception. The reason is simple:

FireDAC’s internal logic depends on TDataSet.FieldDefs, but calclulated fields won’t show up there. You have to use “InternalCalc” instead of “Calculated”, to push your calculated fields to its datataset’s FieldDefs collection.

TTask / TThreadPool may keep interfaced objects alive

UPDATE: The issue described here has been marked “resolved” on May, 5th 2017, so we should see corrected behaviour in one of the next deliveries.

I am using TTask.Run quite a lot in projects and in many of my demos. I won’t go into the details of how to use TTask here, but I only want to highlight that there is still an open issue, which may cause interfaced objects to be destroyed much later, than you would expect. Ultimately this may lead to a much higher memory footprint of your application.

As of Delphi 10.1 Update 2 the issue is still marked open on quality.emabaracdero.com – you might vote for it, if your code is affected by it.

Interfaced objects use automatic reference counting for lifecycle management:

var
  LFoo : IFoo
begin
  LFoo := TFoo.Create;
  //work with LFoo
end;  //LFoo goes out of scope, its reference counter will go to zero,
      // thus it will be destroyed here automatically.

With TTask.Run you can start background threads, that will execute code passed in as anonymous method:

var
  LFoo : IFoo
begin
  LFoo := TFoo.Create;
  TTask.Run(procedure
    begin
      Foobar(LFoo); //Work with LFoo 
    end;
  ).Waitfor; //Wait for task/thread to be completed 
end;  //LFoo goes out of scope, its reference counter SHOULD go to zero,
      // thus it SHOULD be destroyed here automatically.   

Unfortunately the above does not work as expected. The internals of TTask (actually TThreadpool) keep a reference to the (already finished) task and prevent the instance of TFoo, which is referenced by LFoo, to be destroyed automatically. At least when the TThreadPool gets destroyed, all those kept references will be released and the instances will be destroyed, thus they won’t report as memory leak (with ReportMemoryLeaksOnShutDown).

In other words: you will have to set LFoo to nil manually – until this issue gets solved in one of the next updates.

  TTask.Run(procedure
    begin
      Foobar(LFoo); //Work with LFoo 
    end;
  ).Waitfor; //Wait for task/thread to be completed 
  LFoo := nil; //Depending on your logic, this could also be done inside TTask.Run
end;    

Sharing DataModules between VCL and FMX projects

When you start implementing cross-platform applications in Delphi’s FireMonkey framework, then you might consider reusing TDataMdoule instances from your existing VCL applications. If your code is clean enough, i.e. your DataModules are separated from any UI code, that is contain no references to WinApi, Forms, Controls or other UI relevant units, then this is actually a reasonable approach.

You might run into a weird problem though: The IDE will sometimes insist on adding VCL-dependant units to your DataModul, even if you actually compile for FireMonkey. One example is FireDAC.VCL.Wait which will get added automatically under certain conditions (and which won’t compile under FMX).

There is a property that controls this behaviour in shared TDataModule instances:

ClassGroup is basically a “Pseudo Property”, which adds some magic to the project so that the IDE will behave differently depending on the properties value. To make your DataModule really platform independent set it to:   System.Classes.TPersistent

That way, the IDE stops inserting references to either FMX or VCL units in the DataModule’s uses clause. In the case of FireDAC applications, it will just add FireDAC.ConsoleUI.Wait, which will compile just fine on all platforms. You would then add the actual Wait unit in the app’s main form (or in the DPR).

Wir benutzen Cookies um die Nutzerfreundlichkeit der Webseite zu verbessen. Durch Deinen Besuch stimmst Du dem zu.