Skip to main content

Experian Pandora release notes

v5.9.0

May, 2018

We are pleased to announce the release of Pandora 5.9.

The latest version of Pandora groups together several features improving the usability and extensibility of the product. We have added the ability to schedule and load multiple tables, which you can now shuffle to change the loading order. This is a powerful new set of functionality that expands the use cases for Pandora and significantly eases the use of Pandora in large scale migrations and projects. We have also added the ability to call an external web service from within Pandora which means customers can use external web resources to enrich their content.

Full list of new features (details on each available on the Pandora 5.9 documentation pages):

Addition of web service call functionality

The web service call function allows connections to most types of web services enabling users of Experian Pandora to call the web services directly in Pandora. Functionality includes:

  • Results caching to improve performance
  • Exposing cached data for visualisation and searching outside of Experian Pandora.
  • Web services proxy setup in the Pandora GUI
  • Basic and OAuth2.0 (client credentials) authorization support
  • Logging to make it easier to debug

Any Web Service, that returns information in an organised and standardised structure, can be used. Experian Pandora can match JSON, XML, SOAP or Plain Text providing the response can be templated through configuration.

In our documentation, we have included an example web call to What3words that shows how you can pull data in from an external source to enrich your address data.

Project names and source now displayed in Columns and Tables drilldowns

The Columns and Tables drilldowns now have columns displaying a table's projects, schema, source it was loaded from and the source type. This is to help in regulatory reporting where customers may want to be able to export the source and file lineage to data governance.

Ability to create a reusable rules threshold

In previous versions of Pandora, a user would need to create a new rule threshold every time they wanted to validate a rule. A reusable 'Rule Threshold' can now be created and saved under a name, storing a low and high threshold value. For example, a user can create and name thresholds according to the particular quality metric required for a particular regulation.

Low/High Threshold for Rules now displayed in Columns and Tables drilldowns

The Columns and Tables drilldowns now have columns displaying the low and high validation threshold and threshold name for rules.

Ease the process of scheduling automated table loads on a connection

A user can now access a simple self-contained screen within Pandora that allows them to search and filter tables by name and multiselect tables to add them to a scheduled load. This feature vastly improves the usability of this functionality in the product (previously tables had to be manually entered by text). Tables can then be found with minimum effort and added or removed from the schedule as required. In addition to this, a user can now shuffle the order the tables are loaded in to better manage system resources and schedule.

Ability to add prefix/suffix when loading multiple tables

A prefix and suffix to a table name can now be added when configuring one or more tables allowing, for example, users to create a group name or version across several tables. This can also be done for multiple flat files and manual JDBC loads, in the configuration and load dialog.

Ability to map individual columns when exporting via JDBC

In previous versions of Pandora, when exporting data via JDBC, customers would have to map all the columns from their table in Pandora to the source. The target source columns would have to exactly match the columns in the table in Pandora or it would fail.

You can now map individual columns to the target which means you can update single columns. This is a huge improvement in export functionality that will allow clients much more flexibility in using Pandora to export via JDBC.

Rename the "Leading Zero as a number" in Configure Columns menu 

The name of this standardisation option has been changed to "Treat numeric values starting with zero as text (keep leading zeros)" as the original text was not user friendly.

Changes to the status of audit types are logged to the server log

When an audit type is enabled or disabled, an entry in the server log will detail the change. This improves reporting for compliance as previously this could not be tracked in the logs.

Amend defragmentation procedure to cope with external processes interference

If directory rename fails post defragmentation, the product will wait and retry for up to 30 minutes. This allows external processes access time to finish with the newly written files and remove any locks on the directory. We have also improved logging such that each stage of defragmentation is logged on completion (rather than at the end of the whole defragmentation process).

Bug Fixes

  • When connecting to an Oracle database all columns are returned twice on upload
    • Schema setting was not being honoured when reading table layout - corrected.
    • Oracle tables which have a space character in the name will no longer be filtered out.
  • JDBC connection held open continuously
    • Any Pandora JDBC connection will not hold open a DB connection continuously in allowed hours.
    • Only open a connection to the database when a refresh or triggered update happens.
  • Client freezes when refreshing connections with associated drilldowns
    • Renaming a connection causes server exceptions and a client freeze. Prevent renaming a connection. If a name change is absolutely necessary, the connection must be deleted and recreated.
  • Decision functions that use error(s) as conditions ignore them rather than returning an error
    • Using an error as a condition in the decision function would simply be ignored and pass the decision onto the next condition. Now, if Pandora comes across an error, the error is passed through and other conditions ignored.
  • Linking drilldowns don't cancel quickly enough on large links if the user hits back button whilst linking is ongoing
    • Hitting back on a linked drilldown would continue the process in the background for an extended period of time before ending. Now the process is killed instantly if there are no users looking at that specific query.
  • Settings in paths.properties file are not honoured
    • Exporting to a custom export location using keywords in the paths.properties file is now working as expected.
  • Date format not inherited correctly when table created from another table/drilldown
    • When a table is created from another table or drilldown, the date formats are copied from the old table to the new table.
  • Extract Date/Time Element not applying Time Zone offset correctly
    • Date/Time transformation now applies UTC instead of Time Zone.
  • Column Language Configuration not respected at File Preview
    • When parsing dates from string values, they are parsed according to locale.
    • No amendments to date formats made. 
    • Warning message displayed in Preview if the date does not fit into the locate date rules.
  • Cell functions not marked as Dynamic
    • Cell functions (e.g. Cell Values, Count Cells, Average Cells) are now correctly treated as Dynamic and therefore cannot be used in Table Expressions. 

v5.8.0

January, 2018

This release groups together a collection of features improving on the theme of improving Experian Pandora for GDPR requirements.

Features include:

  • GDPR validation expressions
  • Global Batch Address Verification improvements
  • Improvements to the ability to leverage the REST API

These features are important for customers for the following reasons:

  • For those currently using Batch integration with Experian Pandora – the new Batch experience is faster and easier to use
  • Processing time for address verification cut from hours to minutes.
  • Manipulating data post-Batch is faster – providing customers with improved time to value
  • Customers with Personally Identifiable information subject to GDPR now have added expressions and functions that allow them to validate their data and report on data quality more effectively
  • Customers can now use Experian Pandora with third party solutions in a more streamlined way through API improvements
  • We have added commands in the API, and improved its documentation, to make Experian Pandora more intuitive for use with partner applications
  • Improved integration of Experian Pandora into a customer’s automated GDPR workflow decreasing time to value

GDPR validation expressions

  • National Insurance number format validation is now available
  • Dynamic Minimum Age check validation is now available
    • A user can now enter a date of birth in Experian Pandora and this check feature will dynamically inform the user (when the drilldown is refreshed) whether the minimum age has been reached
  • Email address format validation is now available
    • There is now an option to check if the format of an email address conforms to the standard

      Note that this replaces the existing functionality using “Match Expression” and the business constant “Email Address”.

Global Batch Address Verification Improvements

Numerous improvements have been made to the integration with Batch Address Verification, to aid configuration and troubleshooting. Additionally, the Batch process has been sped up and is now caching results, making for a much more responsive experience for users. Data can then be manipulated more quickly after a Batch job is run. We have also added a progress indicator so that customers can see the status of the job.

  • Address Validation within Experian Pandora, details:
    • Missing json files added to aid configuration; json files created for ADS sets – this allows you to create and configure a layout in Experian Pandora rather than having to leave the GUI and directly edit the qaworld.ini file
    • Input field configuration in the “Clean Address Settings” (Home > System > Right Click > Clean Address Settings) is simplified for users
    • Experian Pandora now displays the correct country flags for non-ISO datasets (SGF, GBB etc.) Previously it did not display a flag
    • Allow layout configuration for all data, even if unlicensed, or where the data has expired (customers will not be able to display the data until a license is purchased but they can configure the layout of the address)
    • A user can now configure how often they are alerted to batch data expirations
  • Batch Address Verification data and licence expiry handling
    • All valid address layouts are available for selection (even if the data files have expired) when the Batch feature is used
    • When the Batch feature is used an explicit error is given if the data or license for Batch has expired. This also includes the message "Not Licenced" if the Batch feature is missing from the Experian Pandora license
    • “Repeat Interval for Batch and Disk Warnings" setting default updated from every 10 minutes to 8 Hours and is now fully configurable by customers
    • In the Validate and Enrich dialogue the output lines configuration now displays each element name when there are multiple address elements configured on the same address line, improving ease of use
  • Added a verbose Match Type field
    • A verbose Match Type field is now available under the “Cleaning Result” options in “Available Output Columns” for all layouts
  • You can now create or edit Batch layouts in the Client when connecting to a Linux Pandora server

REST API Enhanced to provide management of JDBC connections

We have improved our API to enable customers and partners to better utilise Experian Pandora outside of the user interface. Connections to the application can now be created and deleted from the REST API. It is also now possible to catalogue the tables available within a connection and to initiate a table load against a connection once established.

Third-party software partners will be able to utilise these new commands to help them to integrate new services with Experian Pandora such as data discovery. Customers will be able to operate Experian Pandora seamlessly from within their own workflow tools.

The following commands have been added:

  • GET object/datasource returns a list of connections
  • GET object/datasource/<name> returns details for connection <name>
  • POST object/datasource/<name> creates connection <name> with the parameters in the request body
  • DELETE object/datasource/<name> deletes connection <name>
  • GET object/datasource/<name>/list catalogues the tables available in connection <name>
  • POST object/datasource/<name>/load loads a table identified in the request body from the connection <name>

We have now created Swagger documentation (as well as our existing user guide) to help customers utilise these features: 

https://www.edq.com/documentation/applications/pandora/rest-api/

Other Enhancements

  • The ability to pre-select current table in the table lookup function
    • When a user is doing a table lookup from “insert custom column”, the table that they navigate from is pre-populated with that table name in the table lookup function box, saving time and effort
  • Metro2: Configurable date handling added. (Metro 2 is a proprietary file format used by financial services companies in the U.S.A. We have added support for ingestion of these files in recent versions of Experian Pandora to support clients using our Data360 solution)
    • Previously the Metro 2 handling defaulted to converting dates in to a readable format. Now the user can configure the date format that they want to import the files in (it now defaults to the raw format of the input files)
  • Enhanced rule validation
    • A feature has been added to ignore null (blank) fields in a rule validation to give a true representation of the quality of a customer’s data without the empty fields being included in the “False” count
  • Create a specific view for the Table Expression drilldown
    • Introduction of a view of table expressions across all version of a given table where previously you could only see a specific table
    • Introduction of a view of table expressions across the latest versions of all tables
  • Updated Progress Hive driver with support for Knox authentication

v5.7.2

November, 2017

A bug was found in versions 5.7.0 and 5.7.1 where large numbers of table version groups were created erroneously in Experian Pandora leading to performance issues. 

Version 5.7.2 introduces the following improvements:

  • Fix to the table version groups bug
  • An automatic repair tool for this bug that will repair affected repositories on install

V5.7.2 replaces 5.7.0 and 5.7.1 which have been withdrawn from download.

v5.7.1

October, 2017

This release of Experian Pandora introduces the following enhancements:

  • Metro2 handling improvements:
    • Improved Metro2™ file handling: all known correct Metro formats are now handled by Experian Pandora
    • Improved Metro2™ error logging: configurable error logging is now available for Metro2™ files and Pandora produces error messages for incorrect Metro2™ file formats. All errors are logged. 
  • Amazon S3 buckets: Experian Pandora can now successfully access files inside folders in Amazon S3 buckets.
  • Improvement to JDBC SQL connection when connecting via Windows Authentication
  • Experian Pandora now handles exporting of an SQL table which has non-alphabetically ordered composite keys

v5.7.0

Show more/hide

 

Aug 30, 2017

This release of Experian Pandora introduces the following enhancements:

  • Support for Metro2 file conversion
  • A globally configurable maximum number of allowed table versions
  • Dates and times consistency when interchanging between databases
  • Logging and user assistance improvements

We've also fixed a number of bugs.

Enhancements

Metro2 File Conversion

It is now possible to convert files from the Metro2 (Packed 366 or Character 426) file formats into a CSV ready for loading into Experian Pandora. Although this functionality is fully tested, test data has been in short supply and it is possible that some aspects of the standard have not been fully catered for. Any such issues will be dealt with should they arise.

Metro2 files should be copied into the drop folder (C:\pandora\data\metro2 by default but this is configurable). They will be automatically converted and, when successful, will be available in Experian Pandora's list of files ready to be loaded. Any errors which occur during the loading process will be logged; successful loads will have the input file removed from the drop directory whilst failures will be left in place. Three attempts will be made to load a file after which time it will be blacklisted until a server restart.

Once converted, records loaded into Experian Pandora will contain a column for all fields of the base record and all fields for each optional segment whether or not they are populated in the source. Whilst there are plans to update the functionality in the future, at present any record containing multiple J1 and J2 segments will see only the last populated instance of each segment written to the destination.

The setting to enable raw format means that numbers dates will not be processed on conversion and will just be stored as a string of digits. This is provided as a diagnostic tool for users who are having difficulty getting their data to convert correctly.

Version management

Better management and visibility of the maximum number of versions that can be kept of each table.

The default value for the maximum number of versions of a table that will be kept remains at 32 but now can be configured by the user. Once set, this default will be applied to all new and existing tables that do not have the maximum individually set. Furthermore, the maximum can now be set across a number of tables at the same time.

It is now possible to view the maximum version count for each table as it appears along with the number of versions actually kept in the table drilldown.

Times and Dates in JDBC

In versions of Experian Pandora prior to v5.6.1, the handling of dates and times when interchanging values between database and Experian Pandora has been inconsistent and unreliable. This is in fact a very complex area because often, it is not clear which timezone a date and time relates to. The design goal for this change has been to hide the complexity of timezones from the user by attempting, as far as possible, to leave displayed times unchanged whilst maintaining the integrity of the data.

Specifically, the following points are now true:

  • JDBC Timezone setting has been removed and is ignored
  • Timezone settings applied to individual JDBC connections are ignored
  • As far as possible, users see times and dates in Experian Pandora as they would in their database
  • Where dates and times in the database have a timezone attached, the value is converted to UTC
  • Experian Pandora stores and displays all dates and times in UTC (or as though they were UTC)
  • Dates and times are written back to the database target as UTC; that should mean they are unchanged 

Logging

When logging load table actions, the repository id and table version are recorded.

Table expression cache start/completed actions are now logged. Log entries include table name, version, id and expression name.

JDBC connection strings will now only be shown in errors when the JDBC Debug option is turned on.

A log message will now appear to show that a period of database disconnection has begun. 

User Assistance

Tooltips have been added and updated on the JDBC connection Automation/Refresh Details tabs. An Experian Pandora documentation page has also been created.

Phone Number Validation

The phone number validation libraries have been updated so validation for international phone numbers should now be more accurate. This requires a full installation (you can't upgrade to this version to get the improved phone validation).

Bug fixes

Scheduled tasks will not restart after a non working day.
Any task not scheduled to run on a working day would fail to run after a non-working day unless other scheduled tasks continued to run in the meantime.

Language settings are incorrectly referenced when loading dates in the format YYYY-mm-dd.
This date format is now invariably parsed as year, month, day regardless of the locale of the source data.

Files with associated DDL load yyyy-mm-dd dates incorrectly, and changing the language does not fix it.
The fix for 40165 also applies in this situation where the file is described by a DDL file.

"Default Standardisation Rules" server setting appears to require a server restart but doesn't prompt.
A server restart is now prompted when this value is edited.

JDBC Connection period does not work
A number of fixes made to ensure that the connection period is respected

Cannot export nulls to sql server varchar/char field
Experian Pandora does not distinguish between nulls and empty strings and attempts to hide the concept from the user. An Experian Pandora null (which is a true null or an empty string) will be written to a database field as null if the field is nullable, or otherwise as an empty string. Experian Pandora strings comprising only spaces are always written as is.
Note that this means that if data is read from, and subsequently written back to, SQL server then any empty strings in nullable fields will be converted to null.

Prevent Ghost Table versions appearing
When the Maximum Table Versions setting is changed for a table, ensure the table information accurately reflects the number of table versions stored.

Disabled Connections allow tables to be loaded and exported to

  • Prevented loads from occurring from disabled connections
  • Prevent drilldown exports to a disabled connection - including loads/exports that are scheduled for later.

JDBC connection refresh does not show new tables for particular database types (e.g. Cassandra & Mongo)
New tables are now detected with a connection refresh

Unable to update records via Experian Pandora for various JDBC Drivers
It is now possible to insert/update/delete records for Mongo and Cassandra databases

JDBC Load Query keeps running when Load job is stopped in Experian Pandora - SQL
When loading a table from a SQLdatabase, if you stop the job on Experian Pandora, the SQL query now stops in the source SQL database.

Port error when attempting to connect to SQLServer Data Direct Driver
This driver now automatically sets the allowPortWithNamedInstance property such that a connection is still made when a named instance connection is supplied as well as a port number.

Client fails to startup on initial attempts if configured database connections time out
Regardless of the connection status of database connections, the client now starts up on the first attempt.

DNS caching never refreshes
The IP address for a hostname is no longer cached indefinitely for database connections so if the IP of a hostname changes, it will now be picked up.

"Drilldowns (All Versions)" drilldown on a table duplicates all drilldown names
Ghost versions of drilldowns are no longer displayed in the "Drilldowns (All Versions)" panel.

v5.6

Show more/hide

The major features in this release are as follows:

  • JDBC Drivers now packaged with Pandora
    • Experian Pandora will now ship with high-performance database drivers to allow out-of-the-box connections to a wide range of common database servers.
    • The new Drivers fully comply with the JDBC 4.0 specification and are fully supported. Pandora users will no longer need to download and configure JDBC drivers provided by database vendors.
  • List of drivers shipped with first version
    • Oracle
    • SQL Server (Supports SQL Server, SQL Azure)
    • DB2
    • MongoDB
    • PostgreSQL
    • Hive (Supports Apache Hadoop Hive, HortonWorks, Cloudera CDH Hive, Amazon EMR Hive, IBM BigInsights Hive, MapR Hive, Pivotal HD Hive)
    • MySQL
    • Greenplum (Supports Pivotal Greenplum, Pivotal HAWQ)
    • Cassandra (Supports Apache Cassandra, DataStax Enterprise)
    • Informix
    • Salesforce (Supports Salesforce.com, Force.com Applications, Financial Force, Veeva CRM)
    • Spark SQL
    • Sybase
    • Progress OpenEdge
    • Oracle Service Cloud
    • Amazon Redshift

v5.5

Show more/hide

The major features in this release are as follows:

  • Business Glossary reporting for Data Governance and Risk
    • The Pandora Data Quality summary reports (column summaries) now automatically associate the data quality scores and cost/risk amounts to each Business Term in the Business Glossary.
    • Column summaries now include the Business Glossary Categories and Terms to which each data column is associated, alongside the existing rule scores and rule categories.

v5.4

Show more/hide

The major features in this release are as follows:

  • UI Responsiveness 
    • UI is more fluid because objects are being returned faster to the Explorer. 
  • Validation Speed Improvements 
    • Reduced the work done to validate by streamlining function invocation and condition evaluation by the query engine and the processing done by the validation processor. 
  • Expression Cache Improvements 
    • Table Expression Cache builders now initialise in groups, controlled by the server setting Maximum Expression Cache Concurrent Builds.
    • Explaining drilldown SQL Queries is faster.
    • Caching means that when an object is requested for the drilldown, it’s only had to be called once for the entire query. 
  • Mapping Enhancements 
    • Improvements to the Mapping dialog and table mapping process

v5.3

Show more/hide

The major features in this release are as follows:

  • MAC Address Selection for Licensing 
    • Ability to specify exactly the MAC address for which to generate a license key (the license will still be valid when running with a different MAC address, provided that the one for which the license was generated is still present on the server). 
  • SSL (Secure Sockets Layer) Support 
    • All communication between the client and server now uses SSL by default.
    • Support for user-supplied certificates. 
  • Foreign File Loading 
    • It is now possible to load files directly from Microsoft Azure Blob and Amazon S3 cloud storage systems. 
  • Expression Editor Overhaul 
    • Significant appearance and usability improvements to the Expression Editor.

v5.2

Show more/hide

The major features in this release are as follows:

  • Scripting Enhancements 
    • A new Save As script-type has been added. This can be used to save a modified drilldown or table as a new table, or new version of an existing table.
    • Scripts can be written using any of the following supported languages:
      • JavaScript
      • Python
      • Ruby
      • Groovy
      • Scala
      • R
      • Template scripts, to help get started with scripting, are available via the Template button in the code editor
      • Script logs can now be viewed in the UI by right-clicking on a script and selecting View Logs
      • Permissions (read, modify, delete, execute) can be controlled on a per-script basis. The Script user capability controls script usage at the Role level
      • Export scripts can now export via script to a JDBC target as well as to files
  • File Configuration Settings 
    • The Character As Alphanumeric server setting now applies to all data formats, not just JDBC
    • The new Strict CSV Processing server setting gives additional control over how files with .csv extension are handled on load. If set to false, .csv files will not be treated as strictly comma separated values
    • The new server setting End Of Line Precedence can be turned on to give newline characters precedence over quote characters and field delimiters when loading a file.
    • There is a new option to change a file’s extension on the server, through the Client. This can be done by right clicking a fixed-length, comma-separated or otherwise delimited file in the Explorer and selecting Change File Format
  • New/Modified Functions 
    • New functions have been added for parsing, formatting and validating international phone numbers
      • Format Phone Number – Takes a column or value as an input and then will either output a correctly formatted phone number, extract information from the phone number (for example the Number Type: Fixed Line, Mobile, Toll-free), or return a Not A Number error.
      • Phone Number Match Code – Compares two values as phone numbers and returns the match code (For example NSN Match, Exact Number, No Match)
      • Phone Numbers Match – Tests if two phone numbers are an exact match for a given country ISO, taking into account valid formats, country codes, etc.
      • Validate Phone Number – Validates a phone number using country-specific validation tests based on a supplied county ISO code.
      • Example Phone Number – Creates an example number for a particular country in a specified format 
    • The Join Lists function has been renamed Combine Lists, and for clarity the various list join operations been renamed as follows:
      • Matched is now Intersection
      • Combine is now Union
      • Right Unmatched is now Remove Left
      • Left Unmatched is now Remove Right
      • Left and Right Unmatched is now Difference

v5.1

Show more/hide

The major features in this release are as follows:

  • Scripting Enhancements 
    • Scripts can now be defined as one of a number of different types:
      • Export - a script that can provide custom export functionality
      • Batch Script - a script that can be executed in a Server job
      • Server - a general Server-side script that can be invoked by the Client
      • Server Model -a script that is available to other Server-side scripts
      • Client UI - a script that can optionally display UI components in a Client desktop window
      • Client Module - a script that is available to other Client-side scripts
      • RESTful - a script that is executed in a web container on the Client 
    • Scripts now support parameters, which can be defined by the user at invoke time.
    • Refer to the Scripting and REST Guide.pdf for full details on the new scripting functionality. 
  • Mapping Dialog Enhancements 
    • The Mapping Dialog (used to map the names of a source drilldown onto a target layout) now persists its state in a drilldown, allowing the incremental development of a source to target mapping and, optionally, the ability to temporarily enable/disable the mapping without losing the mapping state. 
  • File Upload 
    • The process of uploading files to the Server import area has been simplified: clicking on the Upload button on the quick toolbar now immediately shows a file chooser dialog rather than the previous File Manager dialog.
    • Support for uploading metadata files (DDL, COB and XML) to the metadata directory.