Skip to main content

Experian Pandora release notes

Version 5.7.0

Aug 30, 2017

This release of Experian Pandora introduces the following enhancements:

  • Support for Metro2 file conversion
  • A globally configurable maximum number of allowed table versions
  • Dates and times consistency when interchanging between databases
  • Logging and user assistance improvements

We've also fixed a number of bugs.

Enhancements

Metro2 File Conversion

It is now possible to convert files from the Metro2 (Packed 366 or Character 426) file formats into a CSV ready for loading into Experian Pandora. Although this functionality is fully tested, test data has been in short supply and it is possible that some aspects of the standard have not been fully catered for. Any such issues will be dealt with should they arise.

Metro2 files should be copied into the drop folder (C:\pandora\data\metro2 by default but this is configurable). They will be automatically converted and, when successful, will be available in Experian Pandora's list of files ready to be loaded. Any errors which occur during the loading process will be logged; successful loads will have the input file removed from the drop directory whilst failures will be left in place. Three attempts will be made to load a file after which time it will be blacklisted until a server restart.

Once converted, records loaded into Experian Pandora will contain a column for all fields of the base record and all fields for each optional segment whether or not they are populated in the source. Whilst there are plans to update the functionality in the future, at present any record containing multiple J1 and J2 segments will see only the last populated instance of each segment written to the destination.

The setting to enable raw format means that numbers dates will not be processed on conversion and will just be stored as a string of digits. This is provided as a diagnostic tool for users who are having difficulty getting their data to convert correctly.

Version management

Better management and visibility of the maximum number of versions that can be kept of each table.

The default value for the maximum number of versions of a table that will be kept remains at 32 but now can be configured by the user. Once set, this default will be applied to all new and existing tables that do not have the maximum individually set. Furthermore, the maximum can now be set across a number of tables at the same time.

It is now possible to view the maximum version count for each table as it appears along with the number of versions actually kept in the table drilldown.

Times and Dates in JDBC

In versions of Experian Pandora prior to v5.6.1, the handling of dates and times when interchanging values between database and Experian Pandora has been inconsistent and unreliable. This is in fact a very complex area because often, it is not clear which timezone a date and time relates to. The design goal for this change has been to hide the complexity of timezones from the user by attempting, as far as possible, to leave displayed times unchanged whilst maintaining the integrity of the data.

Specifically, the following points are now true:

  • JDBC Timezone setting has been removed and is ignored
  • Timezone settings applied to individual JDBC connections are ignored
  • As far as possible, users see times and dates in Experian Pandora as they would in their database
  • Where dates and times in the database have a timezone attached, the value is converted to UTC
  • Experian Pandora stores and displays all dates and times in UTC (or as though they were UTC)
  • Dates and times are written back to the database target as UTC; that should mean they are unchanged 

Logging

When logging load table actions, the repository id and table version are recorded.

Table expression cache start/completed actions are now logged. Log entries include table name, version, id and expression name.

JDBC connection strings will now only be shown in errors when the JDBC Debug option is turned on.

A log message will now appear to show that a period of database disconnection has begun. 

User Assistance

Tooltips have been added and updated on the JDBC connection Automation/Refresh Details tabs. An Experian Pandora documentation page has also been created.

Phone Number Validation

The phone number validation libraries have been updated so validation for international phone numbers should now be more accurate. This requires a full installation (you can't upgrade to this version to get the improved phone validation).

Bug fixes

Scheduled tasks will not restart after a non working day.
Any task not scheduled to run on a working day would fail to run after a non-working day unless other scheduled tasks continued to run in the meantime.

Language settings are incorrectly referenced when loading dates in the format YYYY-mm-dd.
This date format is now invariably parsed as year, month, day regardless of the locale of the source data.

Files with associated DDL load yyyy-mm-dd dates incorrectly, and changing the language does not fix it.
The fix for 40165 also applies in this situation where the file is described by a DDL file.

"Default Standardisation Rules" server setting appears to require a server restart but doesn't prompt.
A server restart is now prompted when this value is edited.

JDBC Connection period does not work
A number of fixes made to ensure that the connection period is respected

Cannot export nulls to sql server varchar/char field
Experian Pandora does not distinguish between nulls and empty strings and attempts to hide the concept from the user. An Experian Pandora null (which is a true null or an empty string) will be written to a database field as null if the field is nullable, or otherwise as an empty string. Experian Pandora strings comprising only spaces are always written as is.
Note that this means that if data is read from, and subsequently written back to, SQL server then any empty strings in nullable fields will be converted to null.

Prevent Ghost Table versions appearing
When the Maximum Table Versions setting is changed for a table, ensure the table information accurately reflects the number of table versions stored.

Disabled Connections allow tables to be loaded and exported to

  • Prevented loads from occurring from disabled connections
  • Prevent drilldown exports to a disabled connection - including loads/exports that are scheduled for later.

JDBC connection refresh does not show new tables for particular database types (e.g. Cassandra & Mongo)
New tables are now detected with a connection refresh

Unable to update records via Experian Pandora for various JDBC Drivers
It is now possible to insert/update/delete records for Mongo and Cassandra databases

JDBC Load Query keeps running when Load job is stopped in Experian Pandora - SQL
When loading a table from a SQLdatabase, if you stop the job on Experian Pandora, the SQL query now stops in the source SQL database.

Port error when attempting to connect to SQLServer Data Direct Driver
This driver now automatically sets the allowPortWithNamedInstance property such that a connection is still made when a named instance connection is supplied as well as a port number.

Client fails to startup on initial attempts if configured database connections time out
Regardless of the connection status of database connections, the client now starts up on the first attempt.

DNS caching never refreshes
The IP address for a hostname is no longer cached indefinitely for database connections so if the IP of a hostname changes, it will now be picked up.

"Drilldowns (All Versions)" drilldown on a table duplicates all drilldown names
Ghost versions of drilldowns are no longer displayed in the "Drilldowns (All Versions)" panel.

v5.6

Show more/hide

The major features in this release are as follows:

  • JDBC Drivers now packaged with Pandora
    • Experian Pandora will now ship with high-performance database drivers to allow out-of-the-box connections to a wide range of common database servers.
    • The new Drivers fully comply with the JDBC 4.0 specification and are fully supported. Pandora users will no longer need to download and configure JDBC drivers provided by database vendors.
  • List of drivers shipped with first version
    • Oracle
    • SQL Server (Supports SQL Server, SQL Azure)
    • DB2
    • MongoDB
    • PostgreSQL
    • Hive (Supports Apache Hadoop Hive, HortonWorks, Cloudera CDH Hive, Amazon EMR Hive, IBM BigInsights Hive, MapR Hive, Pivotal HD Hive)
    • MySQL
    • Greenplum (Supports Pivotal Greenplum, Pivotal HAWQ)
    • Cassandra (Supports Apache Cassandra, DataStax Enterprise)
    • Informix
    • Salesforce (Supports Salesforce.com, Force.com Applications, Financial Force, Veeva CRM)
    • Spark SQL
    • Sybase
    • Progress OpenEdge
    • Oracle Service Cloud
    • Amazon Redshift

v5.5

Show more/hide

The major features in this release are as follows:

  • Business Glossary reporting for Data Governance and Risk
    • The Pandora Data Quality summary reports (column summaries) now automatically associate the data quality scores and cost/risk amounts to each Business Term in the Business Glossary.
    • Column summaries now include the Business Glossary Categories and Terms to which each data column is associated, alongside the existing rule scores and rule categories.

v5.4

Show more/hide

The major features in this release are as follows:

  • UI Responsiveness 
    • UI is more fluid because objects are being returned faster to the Explorer. 
  • Validation Speed Improvements 
    • Reduced the work done to validate by streamlining function invocation and condition evaluation by the query engine and the processing done by the validation processor. 
  • Expression Cache Improvements 
    • Table Expression Cache builders now initialise in groups, controlled by the server setting Maximum Expression Cache Concurrent Builds.
    • Explaining drilldown SQL Queries is faster.
    • Caching means that when an object is requested for the drilldown, it’s only had to be called once for the entire query. 
  • Mapping Enhancements 
    • Improvements to the Mapping dialog and table mapping process

v5.3

Show more/hide

The major features in this release are as follows:

  • MAC Address Selection for Licensing 
    • Ability to specify exactly the MAC address for which to generate a license key (the license will still be valid when running with a different MAC address, provided that the one for which the license was generated is still present on the server). 
  • SSL (Secure Sockets Layer) Support 
    • All communication between the client and server now uses SSL by default.
    • Support for user-supplied certificates. 
  • Foreign File Loading 
    • It is now possible to load files directly from Microsoft Azure Blob and Amazon S3 cloud storage systems. 
  • Expression Editor Overhaul 
    • Significant appearance and usability improvements to the Expression Editor.

v5.2

Show more/hide

The major features in this release are as follows:

  • Scripting Enhancements 
    • A new Save As script-type has been added. This can be used to save a modified drilldown or table as a new table, or new version of an existing table.
    • Scripts can be written using any of the following supported languages:
      • JavaScript
      • Python
      • Ruby
      • Groovy
      • Scala
      • R
      • Template scripts, to help get started with scripting, are available via the Template button in the code editor
      • Script logs can now be viewed in the UI by right-clicking on a script and selecting View Logs
      • Permissions (read, modify, delete, execute) can be controlled on a per-script basis. The Script user capability controls script usage at the Role level
      • Export scripts can now export via script to a JDBC target as well as to files
  • File Configuration Settings 
    • The Character As Alphanumeric server setting now applies to all data formats, not just JDBC
    • The new Strict CSV Processing server setting gives additional control over how files with .csv extension are handled on load. If set to false, .csv files will not be treated as strictly comma separated values
    • The new server setting End Of Line Precedence can be turned on to give newline characters precedence over quote characters and field delimiters when loading a file.
    • There is a new option to change a file’s extension on the server, through the Client. This can be done by right clicking a fixed-length, comma-separated or otherwise delimited file in the Explorer and selecting Change File Format
  • New/Modified Functions 
    • New functions have been added for parsing, formatting and validating international phone numbers
      • Format Phone Number – Takes a column or value as an input and then will either output a correctly formatted phone number, extract information from the phone number (for example the Number Type: Fixed Line, Mobile, Toll-free), or return a Not A Number error.
      • Phone Number Match Code – Compares two values as phone numbers and returns the match code (For example NSN Match, Exact Number, No Match)
      • Phone Numbers Match – Tests if two phone numbers are an exact match for a given country ISO, taking into account valid formats, country codes, etc.
      • Validate Phone Number – Validates a phone number using country-specific validation tests based on a supplied county ISO code.
      • Example Phone Number – Creates an example number for a particular country in a specified format 
    • The Join Lists function has been renamed Combine Lists, and for clarity the various list join operations been renamed as follows:
      • Matched is now Intersection
      • Combine is now Union
      • Right Unmatched is now Remove Left
      • Left Unmatched is now Remove Right
      • Left and Right Unmatched is now Difference

v5.1

Show more/hide

The major features in this release are as follows:

  • Scripting Enhancements 
    • Scripts can now be defined as one of a number of different types:
      • Export - a script that can provide custom export functionality
      • Batch Script - a script that can be executed in a Server job
      • Server - a general Server-side script that can be invoked by the Client
      • Server Model -a script that is available to other Server-side scripts
      • Client UI - a script that can optionally display UI components in a Client desktop window
      • Client Module - a script that is available to other Client-side scripts
      • RESTful - a script that is executed in a web container on the Client 
    • Scripts now support parameters, which can be defined by the user at invoke time.
    • Refer to the Scripting and REST Guide.pdf for full details on the new scripting functionality. 
  • Mapping Dialog Enhancements 
    • The Mapping Dialog (used to map the names of a source drilldown onto a target layout) now persists its state in a drilldown, allowing the incremental development of a source to target mapping and, optionally, the ability to temporarily enable/disable the mapping without losing the mapping state. 
  • File Upload 
    • The process of uploading files to the Server import area has been simplified: clicking on the Upload button on the quick toolbar now immediately shows a file chooser dialog rather than the previous File Manager dialog.
    • Support for uploading metadata files (DDL, COB and XML) to the metadata directory.

Copyright ©, 2014-2017. All rights reserved.