Datakick - data export and import

Bugfixes

• Fixed bug that prevented data import
• Fixed more PHP 8.2 warnings

New collection - Payments

• You can now access and update order payment informations

PHP8.2 compatibility

Updated module to be compatible with PHP8.2

Bugfixes

• Fixed bug with cron scheduler loop never ending
• Fix problem with temp csv files not being clean after xml generation

New features

• Added new task: Clear datakick executions. This task allows you to clear executions log datakick module tracks. You can run it manually or schedule it to clear logs periodically.

New features

• Added Product: out of stock behaviour field
• Field Attribute::id is now updatable

Bugfixes

• Fix dashboard scroll bug
• Fix import definition opening error
• Fixed bug with calculating required parameters for primary keys
• Fixed bug that prevented importing primary keys
• productImages function now returns images correctly ordered

General code maintenance

Performed code maintenance, improved compatibility with new versions of php and mysql

New features

• function productFeatureValue - this function returns value of product feature, for example productFeatureValue(10, 2) will return value of feature with id 2 for product with id 10
• new collection Invoice - contains data for generated invoices
• new relationship: product accessories -- you can now export and import product accessories as well

Bugfixes

• fixed bug with permission/restriction page
• fixed bug in combination collection
• fixed bug with currency formatting of negative numbers

XML Import is here

Great news! You can now use DataKick to import data to your store

• easily import products, categories, orders, and more
• transform data during the import, for example to automatically apply your profit margin
• you can schedule your import to keep your data in sync with suppliers
• we support huge input files and long running imports

Give it a try, you'll love it. Simply download the new version, or test it on our demo account

I'm very excited to announce release of a new version of DataKick module. And it's not just an ordinary release - this one includes the most requested and anticipated feature so far - data import.

I've put a lot of effort into this release, to make sure this functionality will cover needs of most users. I also tried to keep it as simple as possible, but still flexible and versatile. Believe me, it was a hard task. But I'm sure you will love the result.

Here's the short overview of the import feature

Datasources

you can import data from various sources - upload XML file, file stored on your server, or even from remote server

Supported records

At the moment you can import data only into 12 record types (including products and categories). But don't worry - all record types will be supported soon, in case you will need to import customers or orders.

Import mode

you can choose how to import your data. DataKick supports adding new records, updating records already in your database, or even replace all existing records. And yes, you can specify how to match xml data with data in your database.

Modify data on the fly

you can easily transform the data during the import. This is extremely powerful functionality. You can use it to fix input data, or to apply your own transformations. For example, you can automatically adjust product price by your profit margin

Long running imports

datakick can import very large files. Because of the streaming technology it can even import files that don't fit into server's memory. Import itself can run for a very long time - even hours.

This release represents the major crossroads in DataKick development. From now on, DataKick is no longer free.

New pricing schema

From today, datakick is paid module only. I still believe that merchants should have the option to try and evaluate all the features before purchasing the license, so I added a 14-day free trial period.

There is also a new developer license - if you are running your prestashop on localhost, the trial period is extended to 5 years.

In this release we have focused on usability improvements. We listened to you, our users, and decided to revamp the most challenging sections of application.

Collection selector

DataKick support many record types, and it became hard to choose from them. So we have created brand new record type selector.

When you are about to create new list or mass update, or if you need to join-in a new collection, this new screen will help you quickly and easily choose from available record types. The list is alphabetically ordered and important record types are highlighted

Condition builder

With the new condition builder it's extremely easy to create filters and conditions.

The builder will first present you list of all available fields to choose the one you want to base your condition on.

Once you have selected the field, DataKick will automatically offer you all compatible filter functions based on its data type.

This way, you can create simple conditions quite easily. If you need to create more sophisticated conditions you can always switch to advanced mode and define your condition without any restrictions.

New function and Schema enhancement

There is brand new function available called replaceAccentedChars You can use it to replace accented characters, such as é, č or ü, with their non-accented equivalents.

We have also improved productUrl function - it now accepts second parameter named combinationId. This is an optional parameter. If present, the generated url will point to concrete product combination.

Also we have added new metadata to datakick schema to support selecting foreign keys. For example, default category field in products points to categories. When you use this field in condition, DataKick will help you choose the category - you don't need to remember category id anymore. Neat, isn't it?

This release added support for Prestashop's Advanced Stock Management feature.

You can now access information about your warehouses, stock and stock movements. Also you can also work with supply orders, including order details

Advanced Stock Management deprecation

As you may know, Advanced Stock Management feature has been deprecated for a long time, and now it's not even part of prestashop 1.7.

However, there are still many 1.6 users that depends on this feature. It's actually one of the main reason why so many merchants don't want to upgrade to 1.7 release.

We initially didn't plan to add support for this feature into datakick module, as we thought this feature is dead. But with the advent of thirtybees platform it seems this important functionality will prevail. And we hope datakick can help a bit.

This release brings you the option to translate datakick module to your own language.

Translation process

please use the standard Prestashop translation process

1. Go to Localization > Translation page, and find Modify Translations section
2. choose Installed modules translations as a Type of translation, select your language and theme
3. click on Modify button
4. A page with all modules and their translation strings will appear. Find DataKick module
5. Notice there are two sections - one small with only a few translation strings, and a huge one named js-app-translations with more then 850 strings.
6. fill in your translation to both sections, and save them.

Existing translations

At the moment, DataKick module is translated into following languages

• English (default)
• Czech (100% translated)

This version integrates DataKick custom fields with Prestashop backend forms. You can now directly change custom field values from within your backend forms.

1) Create new custom field inside DataKick module

2) Custom fields will be shown automatically when you edit records

3) Products custom fields are displayed in separate tab

Limitations

Unfortunately not all record types are supported.

At the moment, the only way to edit custom fields for unsupported record types is to use DataKick module itself.

Supported record types

• Address
• Category
• Country
• Currency
• Customer
• Employee
• Customer Groups
• Language
• Manufacturer
• Product
• Role
• Shop
• State
• Supplier
• Tag
• Zone

From this version you can set up a new destination type - Email Place. Every file stored to this place will be automatically send to the associated email address as an attachment.

You can use it with any task that generates files, for example to generate CSV or XML files from templates, or when you create database backup.

Currently supported place types

• Directory on server - allows you to save files on the web server
• FTP server - generated files will be uploaded to ftp server or your choosing
• Email - generated files will be send as an attachment to your email address

This release does not bring you many new and exciting features. Instead, it focus on chores.

Permissions and Restrictions

We have neglected for a long time a multi-user installations. Although this module is intended for power user, we are sure that your employees can benefit from it as well. Because it is so powerful we needed a way to restrict your employees to access certain data only.

With our new permissions and restrictions support you can safely let your employees use this module without fear of accidentally leaking sensitive information about your business.

New task types

To spice up this release a bit, we have also added support for some common prestashop tasks. Newly you can use datakick to trigger these tasks:

• Create database backup
• Refresh currency rates
• Rebuild search index
• Rebuild cache and indexes for Layered navigation block

You can run these tasks on regular basis as a scheduled task, associate endpoint url, or even execute them manually whenever you need.

Database backup task supports places, which means you can store your backup file anywhere on your webserver, or t can be automatically uploaded to your external FTP.

Integration with other modules

This version introduces new prestashop hook datakickExtend that other modules can use to provide additional schema information to DataKick module. This will let DataKick use data stored by these third-party modules.

We have also released free module Price Alert - Follow the Price as a proof of concept.

Example integration code in third-party module:

// module install functions
public function install($delete_params = true) {
  $this->registerHook('datakickExtend');
}

// hook providing schema
public function hookDatakickExtend($params) {
  return array(
    'products' => array(
      'fields' => array(
        'myField' => array(
          'description' => 'Module Field',
          'type' => 'string',
          'sql' => 'ps.module_field'
        )
      )
    )
  );
}

Shopping cart schema is available

New collection Shopping carts has been exposed.

New functions

Also, two new functions were released in this version

• isEmpty - this function return true, if argument is empty / null
• replaceReg - this function performs regular replacement

In this release we have introduced a brand new feature - Mass Updates. You can now easily update multiple records at once. It's an ideal tool to change your product prices, quantities or descriptions. You can also use it to ensure you data quality, for example automatically check and fix common spelling errors.

Mass updates comes with easy to use editor. If you ever created a list you will find it very familiar. You can choose from one of many predefined update operations, such as increate by percentage, replace text or invert boolean values.

If predefined operations are not enough you can always create you very own update expressions

In this minor release we exposed data about your website visitors to DataKick module. From now on you can work with Visitors, Sessions and Page View data.

Also, one new function toUnixTimestamp was created.

In this minor version we have introduced a new function for summarizing attributes - productAttributeValues.

This function can be used for products with combinations. It will return list of all distinct attribute values for single attribute. This functionality is very useful for summarizing product properties - for example to show all existing sized or colors.

This new function was a prerequisite for xml feed for Skroutz comparison service.

In this minor version we have introduced two new functions

• categoryUrl - returns url to category
• productImages - returns list of all image URLs associated with given product

These functions are important in order to create feeds for more comparison services

This release brings you two big new features that makes DataKick module much more powerful and flexible.

Inline editing

You can now use DataKick to change data directly from your lists. If you need to quickly adjust your product quantities, description or price, this functionality is ideal for you. Read more about this feature here.

Custom fields

You can also create new custom fields on standard records, such as products or customers. These fields are great for tracking additional informations. You can, of course, use them in your lists and xml templates.

AdHoc execution

AdHoc execution is the main feature released in this version. This functionality can be used to manually trigger tasks directly from your back office. You can execute any task that could be triggered by Schedule or Endpoint.

• you can manually trigger your scheduled tasks - no need to wait for schedule
• you can download CSV/XML files directly from your endpoint definition
• you will find new action buttons on lists that allows you to export list to CSV, or generate CSV file on your server or FTP

Changes to schema

• new multiple-value fields were added to Product collection - categories, tags, features
• new fields shopId and description are available in Category collection
• new field condition was added to Combination collection
• new expression age was added to Customer collection

New functions

Following new functions were created and can be used in any expression

• categoryImage function returns URL to category image
• dateDiff function returns number of days between two dates

New items in store

I have added two new lists to store. These lists generate CSV files compatible with PrestaShop CSV import functionality. You can use these list to export your data to CSV files and later re-import them to another store.

• Category list - CSV Import format
• Product list - CSV Import format

BugFixes

Some bugs were fixed as well, for example bug in round function, or ability to rename xml nodes.

I hope you will find these changes useful.

This release fixes a serious bug related to saving XML and List templates. Due to MySQL limitations, templates were truncated during save operation if its size exceeded 64k, resulting in irreparable data corruption. After the fix templates can have up to 16M.

Following minor bugs were also fixed:

• bug in productPrice function regarding input parameter validation
• fix sql generation for Has many and belongs to relationships

Note: This release changes database structure of DataKick tables.

This release focus on data exposure. From now on your can access much more data, including customers and orders. Also, some new functions were added.

Exposed records:

• addresses
• carriers
• customers
• customer groups
• orders
• ordered products
• zones
• countries
• states

New functions:

• toCurrency converts numerical value to currency
• floor converts numerical value to currency
• ceil converts numerical value to currency

We are excited to announce release of new major version of this module.

In this release we have added support for Lists and CSV export. From now on, you can use DataKick to create beautiful tabular views of your data. You can use excel-like formulas to format columns, filter rows, or to create advanced highlighting rules.

DataKick comes with very intuitive and easy to use List Builder. Because of its build-in preview functionality you see exactly what data you work with. You need absolutely no coding experience to create a new list. Check this youtube video to see how easy it is to create one.

Of course, you don't have to create lists from scratch. Simply install some predefined template from the store and modify it to your needs.

You can work with lists very easily directly from your back office. But you can get your data out as well. Simply deploy list to endpoint, or create a scheduled task, and you will get the data in CSV format. Ideal for processing in excel.

This release contains few more minor features. Fo example, you will find new dashboard with history of executions - you can easily check when was the last time your scheduled tasks run or see when someone downloads your data via endpoint.

If you encounter any problem or have a suggestion how to make this product better, feel free to contact us at info@getdatakick.com. We are always happy to talk with you guys.

Enjoy!

Export and import your XML Templates

You can now download your XML template definition to your computer and later re-import it.

You can use this mechanism to transfer your configuration between multiple prestashop installations.

Schema improvements

Catalog schema was enhanced so you can easily access more data.

• added new fields to combination collection
• added new relation between feature values and combination collections
• added new relation between manufacturers and combination collections
• added new relation between tags and combination collections

Bugfix - MariaDB

MariaDB version 10.x.x contains bug that manifests when XML template uses array functions tail or init. This release fixes this problem by rewriting SQL.

Schema improvements

• added new fields to combination collection
• added new relation between product and category collections
• added new relation between combination and category collections

New functions were introduces

• trim - remove leading and trailing whitespace
• replace - replace substring by with another string
• substring - returns substring from given position
• toLowerCase - converts input to lower case
• toUpperCase - converts input to upper case
• startsWith - returns true if input strings starts with given prefix
• endsWith - returns true if input string ends with given suffix
• contains - returns true if input string contains given substring
• md5 - returns md5 hash of input
• sha1 - returns sha1 hash of input

Most of the xml templates requires single SQL query to fetch the data. Some more complicated templates, like Google Merchant Center, requires multiple sql queries. In these cases datakick engine run all queries and temporarily store results to a CSV files. After all queries has been executed it merges results into single XML.

Unfortunately there was a bug regarding new lines - if any data contained new line character, parsing CSV files would fail

This version fixes issue with PrestaShop installations running in subdirectory. In this setup module displayed blank page only.

This release fixes several minor issues

This release is all about data exposure. In previous version only a limited subset of records (product, categories) were available for export. Since this version you can export all catalog data:

• products
• categories
• tags
• features
• attributes
• product combinations
• images

We are happy to announce new major version of datakick module.

This release is all about cron and scheduled tasks.

Using this new feature you can generate your XML files on regular basis. Every XML template can have multiple schedules, each can have different settings like output filename, parameter values etc. Generated files can be stored inside any writable directory on your server, uploaded to FTP or stored in Amazon S3 cloud.

Main features

• unlimited number of scheduled tasks
• each schedule provides parameters for xml generation, such as Language
• customizable schedule interval - your scheduled tasks can run hourly, daily, weekly, monthly or even once a year
• save xml files to folders on your webserver
• upload xml files to FTP server
• upload xml files to Amazon S3 cloud
• parametrized file name - for example /categories/{year}-{month}-{day}/category.xml will create file /categories/2017-01-01/category.xml

Use cases

1. pre-populate XML feeds to increase response time
2. schedule regular data backup

Screenshots

Cron settings

Schedule form

Although PHP version 5.4 is not widely used anymore, PrestaShop still officially supports it. We have decided to support for this version as well.

Older PHP versions are not supported.

Since this version you can define custom parameters and used them anywhere in your XML templates. Your XML templates can therefore be much more generic.

Parameters can be used in any expression, including filter expressions. Parameters are replaced by constant values that are provided during runtime, for example via url parameters.

use case

I need XML file containing only products with quantity below some threshold:

1. define custom parameter quantityBelow of type number
2. create product condition product.quantity < runtime.quantityBelow
3. associate xml with url check-quantity, and map quantityBelow parameter to url parameter named below
4. open www.yourshop.com/endpoint/check-quantity?below=100 or www.yourshop.com/endpoint/check-quantity?below=8

The first version of Data Manager Module for Prestashop is released!

This PrestaShop module will help you create and publish XML data feeds directly from your backoffice. Module comes with easy to use visual editor which ensures no coding experience is required on your side.

You can use it to integrate with services like Google Merchant Center, to provide custom data to your affiliate partners, or for periodic backup of your data.

Main features

• Visual XML builder - create valid xml easily using visual composer
• Live data preview - you can see small portion of xml feed directly inside builder
• Formulas - adjust output data using excel-like formulas, for example IF(product.quantity>0, 'In Stock', 'Out Of Stock')
• Conditions - select only subset of data to be exported using condition formulas.
• Parameters - You can create parameters and use them inside formulas and/or conditions. You will need to provide parameter values during xml generation. For example: (product.price * runtime.multiplier)
• URL - associate your xml templates with specific url to provide realtime data
• Predefined templates - module contains sample templates for your convenience (Google Merchant Center, Zbozi.cz, Heureka)
• Supports multiple shops, languages and currencies
• Works on PrestaShop version 1.6 and 1.7

Exposed data

in initial version only category and product data are available. More data will be exposed in future versions.

Screenshots

XML Template editor - visually creation of xml templates

Data Preview - see how your XML Template is transformed to XML data files

Template list - all your templates in one place

Endpoint - assign unique URLs to your xml templates