On Monday 8th August 2022, our CTO, had the privilege to present ETL.NET to the global .NET community on the "On .NET Live" YouTube channel !

September 6, 2022
ETL.NET in a nutshell

ETL.NET is a Mass Data Processing Engine developed to make developers’ lives easier. It was developed by Stéphane Royer – Microsoft Most Valuable Professional (“MVP”) Awarded and was supported and sponsored by FundProcess.

ETL.NET is a framework for .NET that enables effortless implementation of fast, low memory impact and easy to maintain Data processing. All normalizationupsert (update + insert), lookup or join operators considerably reduce the effort for any import and transformation. Everything related to handle tracings and tracking errors is automatically done for developers.

Simply put, ETL.NET is a mix of ETL and LINQ inspiration that fills in the gaps between the world of Business Intelligence and Development, left by other ETLs. Consequently, its flexibility allows to provide more operators than ever.

The previous version of ETL.NET has been downloaded 46,000 times and the latest version has been downloaded 4,000 times on NuGet! Every day, ETL.NET is used to proceed millions of Data.

ETL.NET powering the FundProcess Financial Solution

ETL.NET is particularly well suited to the Financial sector, where very large amounts of complex, unstructured, and fragmented Data need to be processed quickly every day.

As an example of flexibility, FundProcess has integrated an ETL.NET “OCR” operator (Optical Character Recognition) into its Solution. Therefore, the ETL of FundProcess can integrate scanned PDF files like any other Data source file.

The FundProcess’ platform integrates a Macro Editor for encoding:

  • File mapping and Data flows with Finance-specific operators ;
  • All types of queries and Business Features ;
  • Parsing of PDF/XLS/Text/XML documents (e.g., NAV pack, invoices, Limited Partnership Agreements, etc.) ;
  • Data Integrity controls.

The above features allow you to obtain a homogenous Database and an auditable Dataset. The aim is to eliminate manual operations and mitigate the operational and security risks of using an ETL or OCR not integrated into the FundProcess platform.

How ETL.NET works
  • Powered by & for .NET

ETL.NET is written entirely in .NET for a multi-platform use and for an easy and straightforward integration into any application. Extending it literally takes 5 minutes.

  • Easy to implement

ETL.NET works with a similar principle to SSIS, with ETL processes to be written in .NET as LINQ queries. Previously, it took a few days to build a proper ETL to import, for instance, CSV files.

  • Easy to run

A simple and straightforward ELT.NET runtime for .NET executes ETL processes with no installation required.

  • Read or write following file types and Data sources
    • Native SQL server
    • Entity Framework
    • CSV
    • Excel
    • Bloomberg response files
    • Scanned PDF
    • XML
    • Anything .NET can read or write whatsoever
  • Read or write files on the following sources
    • File system
    • FTP
    • SFTP
    • FTPS
    • Dropbox
    • eMail and MailBox
    • Zip archives
    • Anything .NET can access whatsoever

Although SSIS is a fantastic tool, ETL.NET offers the following distinguish features:

    • It is actually multiplatform

ETL.NET can be run on almost any kind of computer or even portable devices.

Development is made on .NET. This allows ETL.NET to be used on any platform that supports .NET, i.e., Windows, Linux or MacOS.

    • It is open source

ETL.NET enables you to benefit from the advantages of the open-source community.

Anyone can make a new enhancement request so that developers can implement it for you. You can even add your own amendments to be considered as new features in the next release(s). Of course, open source implies full transparency about what ETL.NET does under the hood.

    • It is made for developers

Outside of a Business Intelligence context, SSIS can be very cumbersome.

When it comes to including ETL process in an application, many developers do not even consider using Business Intelligence tools such as SSIS. This is quite understandable given the significant commitment it involves:

      • SSIS packages are in a different location than the application itself. They are no in the application sources, they will not be deployed in the same way, they will not be deployed in the same place ;
      • Executing the full solution on a development computer for debugging or even simple development purposes is complex. Usually, in the development world, once an application’s source is extracted from source control, hitting F5 should be enough to successfully execute and debug the application. With an SSIS process as part of the solution, executing, and debugging the application requires an SSIS deployment with all its setups before running it. While debugging the application itself, it will not be possible to debug the ETL process as it is hosted in SSIS server ;
      • SSIS must be installed on development computers. This makes the solution more complex to apprehend where normal applications simply require dotnet core + visual studio code or visual studio community to be executed with simply hitting F5. Furthermore, depending on the context, installing SSIS on a development computer is not always easy or possible (corporate security policies).

With ETL.NET, a regular developer has to download his sources, hit F5, and debug.

    • It is really simple and easy

Simply add references to NuGet*.

Just add a NuGet reference to the core package (ETL.NET) and to some of its extensions as needed (e.g., “ETL.NET.TextFile”, “ETL.NET.EntityFrameworkCore”, etc.) and you can develop and execute right away. It is a very simple development and extremely similar to LINQ.

*Microsoft Global repository of shared .NET libraries (commercial or open source).

    • It is designed to be very simple to extend

Building SSIS extensions is a horribly complex process compared to usual .NET toolkits and frameworks.

An ETL.NET extension can be developed, debugged, and used within literally 15 minutes.

ETL.NET’s extensibility is based on a simple development, whether it is integrated into the application, into another shared assembly or into a NuGet package. Only a few lines of code are required a set of base classes allows the easy implementation of extensions for most typical use cases.

Know more about ETL.NET
  • Watch the video broadcasted on 8th August 2022 on the “On .NET Live” YouTube channel

In this video, you will:

    • Watch a presentation of ETL.NET by its developer, Stéphane Royer
    • Watch a “live” demonstration of how the tool is used in a Financial context
    • Get answers to questions such as, why ETL.NET compared to SISS, Rhino ETL, Spark, etc. ?, is ETL.NET can be used through Web API which will take collection as input ?, Why is ETL.NET not a “Visual tool” ?

Click here to watch the Video: https://www.youtube.com/watch?v=ivts2qvSats

  • Go to the ETL.NET GitHub page created by Stéphane Royer

On this GitHub page, you will:

    • Get a detailed description of what ETL.NET is
    • Get the full documentation (always under improvement) allowing you to:
      • Get started
      • Install and execute ETL.NET
      • Get a tutorial
      • Get patterns and recipes
      • Get operators

Click here to access to ETL.NET GitHub page: https://paillave.github.io/Etl.Net/