Skip to main content

Experian Pandora release notes

v5.8.0

January, 2018

This release groups together a collection of features improving on the theme of improving Experian Pandora for GDPR requirements.

Features include:

  • GDPR validation expressions
  • Global Batch Address Verification improvements
  • Improvements to the ability to leverage the REST API

These features are important for customers for the following reasons:

  • For those currently using Batch integration with Experian Pandora – the new Batch experience is faster and easier to use
  • Processing time for address verification cut from hours to minutes.
  • Manipulating data post-Batch is faster – providing customers with improved time to value
  • Customers with Personally Identifiable information subject to GDPR now have added expressions and functions that allow them to validate their data and report on data quality more effectively
  • Customers can now use Experian Pandora with third party solutions in a more streamlined way through API improvements
  • We have added commands in the API, and improved its documentation, to make Experian Pandora more intuitive for use with partner applications
  • Improved integration of Experian Pandora into a customer’s automated GDPR workflow decreasing time to value

REST API Enhanced to provide management of JDBC connections

We have improved our API to enable customers and partners to better utilise Experian Pandora outside of the user interface. Connections to the application can now be created and deleted from the REST API. It is also now possible to catalogue the tables available within a connection and to initiate a table load against a connection once established.

Third-party software partners will be able to utilise these new commands to help them to integrate new services with Experian Pandora such as data discovery. Customers will be able to operate Experian Pandora seamlessly from within their own workflow tools.

The following commands have been added:

  • GET object/datasource returns a list of connections
  • GET object/datasource/<name> returns details for connection <name>
  • POST object/datasource/<name> creates connection <name> with the parameters in the request body
  • DELETE object/datasource/<name> deletes connection <name>
  • GET object/datasource/<name>/list catalogues the tables available in connection <name>
  • POST object/datasource/<name>/load loads a table identified in the request body from the connection <name>

We have now created Swagger documentation (as well as our existing user guide) to help customers utilise these features: 

https://www.edq.com/documentation/applications/pandora/rest-api/

Batch Address Verification Improvements

Numerous improvements have been made to the integration with Batch Address Verification, to aid configuration and troubleshooting. Additionally, the Batch process has been sped up and is now caching results, making for a much more responsive experience for users. Data can then be manipulated more quickly after a Batch job is run. We have also added a progress indicator so that customers can see the status of the job.

  • Address Validation within Experian Pandora, details:
    • Missing json files added to aid configuration; json files created for ADS sets – this allows you to create and configure a layout in Experian Pandora rather than having to leave the GUI and directly edit the qaworld.ini file
    • Input field configuration in the “Clean Address Settings” (Home > System > Right Click > Clean Address Settings) is simplified for users
    • Experian Pandora now displays the correct country flags for non-ISO datasets (SGF, GBB etc.) Previously it did not display a flag
    • Allow layout configuration for all data, even if unlicensed, or where the data has expired (customers will not be able to display the data until a license is purchased but they can configure the layout of the address)
    • A user can now configure how often they are alerted to batch data expirations
  • Batch Address Verification data and licence expiry handling
    • All valid address layouts are available for selection (even if the data files have expired) when the Batch feature is used
    • When the Batch feature is used an explicit error is given if the data or license for Batch has expired. This also includes the message "Not Licenced" if the Batch feature is missing from the Experian Pandora license
    • “Repeat Interval for Batch and Disk Warnings" setting default updated from every 10 minutes to 8 Hours and is now fully configurable by customers
    • In the Validate and Enrich dialogue the output lines configuration now displays each element name when there are multiple address elements configured on the same address line, improving ease of use
  • Added a verbose Match Type field
    • A verbose Match Type field is now available under the “Cleaning Result” options in “Available Output Columns” for all layouts
  • You can now create or edit Batch layouts in the Client & Unix server
  • National Insurance number format validation is now available
  • Dynamic Minimum Age check validation is now available
    • A user can now enter a date of birth in Experian Pandora and this check feature will dynamically inform the user (when the drilldown is refreshed) whether the minimum age has been reached
  • Ability to generate a UUID is available in Pandora
  • Email address format validation is now available
    • There is now an option to check if the format of an email address conforms to the standard

Other Enhancements

  • The ability to pre-select current table in the table lookup function
    • When a user is doing a table lookup from “insert custom column”, the table that they navigate from is pre-populated with that table name in the table lookup function box, saving time and effort
  • Metro2: Configurable date handling added. (Metro 2 is a proprietary file format used by financial services companies in the U.S.A. We have added support for ingestion of these files in recent versions of Experian Pandora to support clients using our Data360 solution)
    • Previously the Metro 2 handling defaulted to converting dates in to a readable format. Now the user can configure the date format that they want to import the files in (it now defaults to the raw format of the input files)
  • Enhanced rule validation
    • A feature has been added to ignore null (blank) fields in a rule validation to give a true representation of the quality of a customer’s data without the empty fields being included in the “False” count
  • Create a specific view for the Table Expression drilldown
    • Introduction of a view of table expressions across all version of a given table where previously you could only see a specific table
    • Introduction of a view of table expressions across the latest versions of all tables
  • Updated Progress Hive driver with support for Knox authentication

v5.7.2

November, 2017

A bug was found in versions 5.7.0 and 5.7.1 where large numbers of table version groups were created erroneously in Experian Pandora leading to performance issues. 

Version 5.7.2 introduces the following improvements:

  • Fix to the table version groups bug
  • An automatic repair tool for this bug that will repair affected repositories on install

V5.7.2 replaces 5.7.0 and 5.7.1 which have been withdrawn from download.

v5.7.1

October, 2017

This release of Experian Pandora introduces the following enhancements:

  • Metro2 handling improvements:
    • Improved Metro2™ file handling: all known correct Metro formats are now handled by Experian Pandora
    • Improved Metro2™ error logging: configurable error logging is now available for Metro2™ files and Pandora produces error messages for incorrect Metro2™ file formats. All errors are logged. 
  • Amazon S3 buckets: Experian Pandora can now successfully access files inside folders in Amazon S3 buckets.
  • Improvement to JDBC SQL connection when connecting via Windows Authentication
  • Experian Pandora now handles exporting of an SQL table which has non-alphabetically ordered composite keys

v5.7.0

Show more/hide

 

Aug 30, 2017

This release of Experian Pandora introduces the following enhancements:

  • Support for Metro2 file conversion
  • A globally configurable maximum number of allowed table versions
  • Dates and times consistency when interchanging between databases
  • Logging and user assistance improvements

We've also fixed a number of bugs.

Enhancements

Metro2 File Conversion

It is now possible to convert files from the Metro2 (Packed 366 or Character 426) file formats into a CSV ready for loading into Experian Pandora. Although this functionality is fully tested, test data has been in short supply and it is possible that some aspects of the standard have not been fully catered for. Any such issues will be dealt with should they arise.

Metro2 files should be copied into the drop folder (C:\pandora\data\metro2 by default but this is configurable). They will be automatically converted and, when successful, will be available in Experian Pandora's list of files ready to be loaded. Any errors which occur during the loading process will be logged; successful loads will have the input file removed from the drop directory whilst failures will be left in place. Three attempts will be made to load a file after which time it will be blacklisted until a server restart.

Once converted, records loaded into Experian Pandora will contain a column for all fields of the base record and all fields for each optional segment whether or not they are populated in the source. Whilst there are plans to update the functionality in the future, at present any record containing multiple J1 and J2 segments will see only the last populated instance of each segment written to the destination.

The setting to enable raw format means that numbers dates will not be processed on conversion and will just be stored as a string of digits. This is provided as a diagnostic tool for users who are having difficulty getting their data to convert correctly.

Version management

Better management and visibility of the maximum number of versions that can be kept of each table.

The default value for the maximum number of versions of a table that will be kept remains at 32 but now can be configured by the user. Once set, this default will be applied to all new and existing tables that do not have the maximum individually set. Furthermore, the maximum can now be set across a number of tables at the same time.

It is now possible to view the maximum version count for each table as it appears along with the number of versions actually kept in the table drilldown.

Times and Dates in JDBC

In versions of Experian Pandora prior to v5.6.1, the handling of dates and times when interchanging values between database and Experian Pandora has been inconsistent and unreliable. This is in fact a very complex area because often, it is not clear which timezone a date and time relates to. The design goal for this change has been to hide the complexity of timezones from the user by attempting, as far as possible, to leave displayed times unchanged whilst maintaining the integrity of the data.

Specifically, the following points are now true:

  • JDBC Timezone setting has been removed and is ignored
  • Timezone settings applied to individual JDBC connections are ignored
  • As far as possible, users see times and dates in Experian Pandora as they would in their database
  • Where dates and times in the database have a timezone attached, the value is converted to UTC
  • Experian Pandora stores and displays all dates and times in UTC (or as though they were UTC)
  • Dates and times are written back to the database target as UTC; that should mean they are unchanged 

Logging

When logging load table actions, the repository id and table version are recorded.

Table expression cache start/completed actions are now logged. Log entries include table name, version, id and expression name.

JDBC connection strings will now only be shown in errors when the JDBC Debug option is turned on.

A log message will now appear to show that a period of database disconnection has begun. 

User Assistance

Tooltips have been added and updated on the JDBC connection Automation/Refresh Details tabs. An Experian Pandora documentation page has also been created.

Phone Number Validation

The phone number validation libraries have been updated so validation for international phone numbers should now be more accurate. This requires a full installation (you can't upgrade to this version to get the improved phone validation).

Bug fixes

Scheduled tasks will not restart after a non working day.
Any task not scheduled to run on a working day would fail to run after a non-working day unless other scheduled tasks continued to run in the meantime.

Language settings are incorrectly referenced when loading dates in the format YYYY-mm-dd.
This date format is now invariably parsed as year, month, day regardless of the locale of the source data.

Files with associated DDL load yyyy-mm-dd dates incorrectly, and changing the language does not fix it.
The fix for 40165 also applies in this situation where the file is described by a DDL file.

"Default Standardisation Rules" server setting appears to require a server restart but doesn't prompt.
A server restart is now prompted when this value is edited.

JDBC Connection period does not work
A number of fixes made to ensure that the connection period is respected

Cannot export nulls to sql server varchar/char field
Experian Pandora does not distinguish between nulls and empty strings and attempts to hide the concept from the user. An Experian Pandora null (which is a true null or an empty string) will be written to a database field as null if the field is nullable, or otherwise as an empty string. Experian Pandora strings comprising only spaces are always written as is.
Note that this means that if data is read from, and subsequently written back to, SQL server then any empty strings in nullable fields will be converted to null.

Prevent Ghost Table versions appearing
When the Maximum Table Versions setting is changed for a table, ensure the table information accurately reflects the number of table versions stored.

Disabled Connections allow tables to be loaded and exported to

  • Prevented loads from occurring from disabled connections
  • Prevent drilldown exports to a disabled connection - including loads/exports that are scheduled for later.

JDBC connection refresh does not show new tables for particular database types (e.g. Cassandra & Mongo)
New tables are now detected with a connection refresh

Unable to update records via Experian Pandora for various JDBC Drivers
It is now possible to insert/update/delete records for Mongo and Cassandra databases

JDBC Load Query keeps running when Load job is stopped in Experian Pandora - SQL
When loading a table from a SQLdatabase, if you stop the job on Experian Pandora, the SQL query now stops in the source SQL database.

Port error when attempting to connect to SQLServer Data Direct Driver
This driver now automatically sets the allowPortWithNamedInstance property such that a connection is still made when a named instance connection is supplied as well as a port number.

Client fails to startup on initial attempts if configured database connections time out
Regardless of the connection status of database connections, the client now starts up on the first attempt.

DNS caching never refreshes
The IP address for a hostname is no longer cached indefinitely for database connections so if the IP of a hostname changes, it will now be picked up.

"Drilldowns (All Versions)" drilldown on a table duplicates all drilldown names
Ghost versions of drilldowns are no longer displayed in the "Drilldowns (All Versions)" panel.

v5.6

Show more/hide

The major features in this release are as follows:

  • JDBC Drivers now packaged with Pandora
    • Experian Pandora will now ship with high-performance database drivers to allow out-of-the-box connections to a wide range of common database servers.
    • The new Drivers fully comply with the JDBC 4.0 specification and are fully supported. Pandora users will no longer need to download and configure JDBC drivers provided by database vendors.
  • List of drivers shipped with first version
    • Oracle
    • SQL Server (Supports SQL Server, SQL Azure)
    • DB2
    • MongoDB
    • PostgreSQL
    • Hive (Supports Apache Hadoop Hive, HortonWorks, Cloudera CDH Hive, Amazon EMR Hive, IBM BigInsights Hive, MapR Hive, Pivotal HD Hive)
    • MySQL
    • Greenplum (Supports Pivotal Greenplum, Pivotal HAWQ)
    • Cassandra (Supports Apache Cassandra, DataStax Enterprise)
    • Informix
    • Salesforce (Supports Salesforce.com, Force.com Applications, Financial Force, Veeva CRM)
    • Spark SQL
    • Sybase
    • Progress OpenEdge
    • Oracle Service Cloud
    • Amazon Redshift

v5.5

Show more/hide

The major features in this release are as follows:

  • Business Glossary reporting for Data Governance and Risk
    • The Pandora Data Quality summary reports (column summaries) now automatically associate the data quality scores and cost/risk amounts to each Business Term in the Business Glossary.
    • Column summaries now include the Business Glossary Categories and Terms to which each data column is associated, alongside the existing rule scores and rule categories.

v5.4

Show more/hide

The major features in this release are as follows:

  • UI Responsiveness 
    • UI is more fluid because objects are being returned faster to the Explorer. 
  • Validation Speed Improvements 
    • Reduced the work done to validate by streamlining function invocation and condition evaluation by the query engine and the processing done by the validation processor. 
  • Expression Cache Improvements 
    • Table Expression Cache builders now initialise in groups, controlled by the server setting Maximum Expression Cache Concurrent Builds.
    • Explaining drilldown SQL Queries is faster.
    • Caching means that when an object is requested for the drilldown, it’s only had to be called once for the entire query. 
  • Mapping Enhancements 
    • Improvements to the Mapping dialog and table mapping process

v5.3

Show more/hide

The major features in this release are as follows:

  • MAC Address Selection for Licensing 
    • Ability to specify exactly the MAC address for which to generate a license key (the license will still be valid when running with a different MAC address, provided that the one for which the license was generated is still present on the server). 
  • SSL (Secure Sockets Layer) Support 
    • All communication between the client and server now uses SSL by default.
    • Support for user-supplied certificates. 
  • Foreign File Loading 
    • It is now possible to load files directly from Microsoft Azure Blob and Amazon S3 cloud storage systems. 
  • Expression Editor Overhaul 
    • Significant appearance and usability improvements to the Expression Editor.

v5.2

Show more/hide

The major features in this release are as follows:

  • Scripting Enhancements 
    • A new Save As script-type has been added. This can be used to save a modified drilldown or table as a new table, or new version of an existing table.
    • Scripts can be written using any of the following supported languages:
      • JavaScript
      • Python
      • Ruby
      • Groovy
      • Scala
      • R
      • Template scripts, to help get started with scripting, are available via the Template button in the code editor
      • Script logs can now be viewed in the UI by right-clicking on a script and selecting View Logs
      • Permissions (read, modify, delete, execute) can be controlled on a per-script basis. The Script user capability controls script usage at the Role level
      • Export scripts can now export via script to a JDBC target as well as to files
  • File Configuration Settings 
    • The Character As Alphanumeric server setting now applies to all data formats, not just JDBC
    • The new Strict CSV Processing server setting gives additional control over how files with .csv extension are handled on load. If set to false, .csv files will not be treated as strictly comma separated values
    • The new server setting End Of Line Precedence can be turned on to give newline characters precedence over quote characters and field delimiters when loading a file.
    • There is a new option to change a file’s extension on the server, through the Client. This can be done by right clicking a fixed-length, comma-separated or otherwise delimited file in the Explorer and selecting Change File Format
  • New/Modified Functions 
    • New functions have been added for parsing, formatting and validating international phone numbers
      • Format Phone Number – Takes a column or value as an input and then will either output a correctly formatted phone number, extract information from the phone number (for example the Number Type: Fixed Line, Mobile, Toll-free), or return a Not A Number error.
      • Phone Number Match Code – Compares two values as phone numbers and returns the match code (For example NSN Match, Exact Number, No Match)
      • Phone Numbers Match – Tests if two phone numbers are an exact match for a given country ISO, taking into account valid formats, country codes, etc.
      • Validate Phone Number – Validates a phone number using country-specific validation tests based on a supplied county ISO code.
      • Example Phone Number – Creates an example number for a particular country in a specified format 
    • The Join Lists function has been renamed Combine Lists, and for clarity the various list join operations been renamed as follows:
      • Matched is now Intersection
      • Combine is now Union
      • Right Unmatched is now Remove Left
      • Left Unmatched is now Remove Right
      • Left and Right Unmatched is now Difference

v5.1

Show more/hide

The major features in this release are as follows:

  • Scripting Enhancements 
    • Scripts can now be defined as one of a number of different types:
      • Export - a script that can provide custom export functionality
      • Batch Script - a script that can be executed in a Server job
      • Server - a general Server-side script that can be invoked by the Client
      • Server Model -a script that is available to other Server-side scripts
      • Client UI - a script that can optionally display UI components in a Client desktop window
      • Client Module - a script that is available to other Client-side scripts
      • RESTful - a script that is executed in a web container on the Client 
    • Scripts now support parameters, which can be defined by the user at invoke time.
    • Refer to the Scripting and REST Guide.pdf for full details on the new scripting functionality. 
  • Mapping Dialog Enhancements 
    • The Mapping Dialog (used to map the names of a source drilldown onto a target layout) now persists its state in a drilldown, allowing the incremental development of a source to target mapping and, optionally, the ability to temporarily enable/disable the mapping without losing the mapping state. 
  • File Upload 
    • The process of uploading files to the Server import area has been simplified: clicking on the Upload button on the quick toolbar now immediately shows a file chooser dialog rather than the previous File Manager dialog.
    • Support for uploading metadata files (DDL, COB and XML) to the metadata directory.