Parquet Source Component is SSIS Data Flow Component for retrieving data from Apache Parquet file that supports multiple outputs via the composite records pattern.
- Supports reading the Apache Parquet files.
- Component metadata is automatically retrieved from the provided Parquet file.
- Supports composite outputs. Besides the root Parquet Source Output that contains the top-level fields, for any nested arrays, corresponding composite outputs get populated.
- Supports an error output for redirecting problematic records (in case of a failure processing the field values).
In this section we will show you how to set up a Parquet Source component.
- Double-click on the component on the canvas.
- Once the component editor opens, select the fIle connection from File Connection menu. If the file is in the correct format, its data schema will be visualized in the Schema text editor. Note that the schema can be entered manually in the editor and the corresponding metadata will be generated even without a file currently available.
- When clicking on Columns tab the component would prepare the outputs and external columns by analyzing the existing data in the Schema text editor. Please note that the Parquet Source can have multiple outputs (see the article about composite records), which columns you can see. The data in these outputs can be processed by downstream transformation and destination components(e.g. multiple OLE DB Destinations can store the data in SQL Server database).
- Click OK to close the component editor.
Congratulations! You have successfully configured the Parquet Source component.
Use the parameters below to configure the component.
Select an existing File Connection Manager or create new.
- Fixed: Incorrect lower-case headers (Thank you, Romain).
- Fixed: Failed with error "System.IO.EndOfStreamException: Unable to read beyond the end of the stream." (Thank you, Naveen).
- New: Considerable performance improvements.
- Fixed: Data mismatch when reading from certain files (Thank you, Jessica).
- New: Introduced component.