Posted on

Pro Pack Gets Faster CSV Import Option

The latest version of the Pro Pack, 4.2.07, has been released with some new features including a “Recode All Uncoded” and “Load Data CSV Import”.

Pro 4.2.07 Bulk Actions
Pro 4.2.07 Bulk Actions

Recode All Uncoded

The Recode All Uncoded bulk action under Manage Locations will automatically select all of the uncoded (missing latitude and/or longitude coordinates) locations without the need to check the boxes.    The process will automatically select all of the uncoded locations and re-submit them to Google for geocoding.

This will not magically fix addresses that Google cannot find.   It is meant to assist sites that have more locations than the daily Google geocoding limit will support.  The free Google Maps API limits location geocoding requests to 2500 per day per server.  On a shared hosting server this can be a few hundred location “lookups” if other sites on the same server are using Google Maps.    Recode All Uncoded will help quickly re-run locations that were previously imported but exceeded the geocoding limit for the day.

Load Data CSV Import

from the updated CSV Import documentation:

If you have Pro Pack version 4.2.06+ and Store Locator Plus version 4.2.41+ you can use the Load Data option with CSV Import.  This feature uses the MySQL Load Data command is is 10-50x faster than WordPress / PHP CSV file parsing, with the typical Google Geocoding limitations on performance and record counts if you are not supplying the latitude/longitude data.   This feature will only import basic location data.  It does not import extended data fields or Tagalong category data.     If you use this along with having pre-entered latitude and longitude values you can import 100,000 locations in less than 10 minutes on a basic web server.    The column headers should be included in the file and should match the basic fields.  You do NOT need to include all columns.

You can build a CSV-import ready export directly from MySQL if your MySQL user account has the GRANT FILE privilege on the WordPress Store Locator Plus file.    You’ll typically need to login to MySQL from an administrative account to grant file to the WordPress or other user as follows:


#mysql

> GRANT FILE ON wordpress.* TO 'wpuser'@'localhost';

You will replace wpuser with your WordPress username and ‘localhost’ with your MySQL server hostname or IP address.

With the file privilege set you can then export a CSV file for import with the following MySQL command:


SELECT * INTO OUTFILE 'geocoded_locations.csv'

FIELDS TERMINATED BY ','

OPTIONALLY ENCLOSED BY '"'

LINES TERMINATED BY '\r\n'

FROM wp_store_locator;

This will create a geocoded_locations.csv file in the default MySQL WordPress files location, typically /var/lib/mysql/wordpress/ on a Linux system.    You can specify a fully-qualified URL for the output file.  The details on where the file is written by default and how to specify the full URL will depend on your web server settings.

PRO 4.2.06 Load Data Import
PRO 4.2.06 Load Data Import

You can now import the basic file using the Store Locator Plus Pro Pack by using the Load Data option with “first line has field names” and “skip geocoding” if you have the following CSV file header:


sl_id,sl_store,sl_address,sl_address2,sl_city,sl_state,sl_zip,sl_country,sl_latitude,sl_longitude,sl_tags,sl_description,sl_email,sl_url,sl_hours,sl_phone,sl_fax,sl_image,sl_private,sl_neat_title,ignore_sl_linked_postid,ignore_sl_pages_url,ignore_sl_pages_on,ignore_sl_option_value,ignore_sl_lastupdated

This will import the primary fields and ignore the secondary data fields that you will want to have re-created automatically.

Export All Hosted CSV

A new experimental feature has also been added.    It is deemed “experimental” as it works on some systems but will not work on all systems due to directory security limitations.  If this feature gets a lot of interest from the users it will be refined and kept in the product. Export All, Hosted CSV is a new CSV export feature that works much like the Export Locations download feature.

The Export Locations feature that existed in the past, and is now labelled “Export All, Download CSV” fetches the location data in CSV format and sends it to the browser for immediate download.  This can be a time consuming process as not only does the server need to process the location list and write the data, but your browser needs to open up a local file on your device and write the data locally.    Depending on how much memory you have on your device and the speed of your local drive the direct download process can be slower.

The Export All, Hosted CSV option keeps everything on the server.    Your browser connection will remain active as Store Locator Plus reads all of the location records on the server and writes them to a disk file on the same server, however there is no browser download.  For SOME users this will be faster, especially if you are on an older laptop or desktop.    The file is written the the WordPress temporary storage directory on the server.   For most Linux based hosts that will be in the /tmp directory which you may not have access to depending on your host.   Some hosts will put temporary files in a local folder under your hosting account with the file name slplus_locations.csv.     If there is enough interest, this option will be refined to allow users to specify the destination directory and filename.      Site administrators can access the local server file using their hosting solution file manager or standard FTP/SFTP applications, which can be faster than browser-based downloads.

Pro Pack Change Log

Posted on

Store Locator Plus Improves High Volume Site Processing

Store Locator Plus 4.2.41 focuses on speeding up the performance of location import processing for comma separated files of 50,000 locations or more.

CSV Import in Add Mode

Some of the updates include reduce execution time when setting the CSV import duplicates processing to ‘add’ mode.    Add mode is useful when importing to a “clean” list of locations.    This mode eliminated extra processing when you know you are not updating existing locations during a Pro Pack CSV Import.

Improved Import Error Reporting

The CSV Import processing system has been updated to provide better reporting when CSV imports of location data via Pro Pack or category data via Tagalong fails the import process.   Various file processing issues as well as memory restriction errors are reported with details including hints as to how to manage the issue.

Memory Improvements

Memory consumption is reduced on per-process system calls.  This reduction can be significant on systems that are running large location lists, saving up to 1k per location processing request.

Store Locator Plus Change Log

Posted on

Janitor 4.1.15 Makes Quick Work Of Resetting Locations

WordPress Locations Tabbar Banner

The Janitor add-on pack for Store Locator Plus adds a new drop locations feature.    This feature makes quick work of deleting the Store Locator Plus locations when a simple database reset is required.   The new feature is far faster than the previous Clear Locations option, but Drop Locations does have its limits.

J 4.2.15 Clear or Drop Locations
J 4.2.15 Clear or Drop LocationsWo

Drop Locations

This new feature removes the main locations table and the extended meta data fields, used by the * Extender add-on packs, via a the SQL DROP TABLE command.  Both tables are immediately recreated.     This feature does NOT remove the associates custom WordPress “store pages” data NOR does it remove the Tagalong and custom taxonomy data.    If you are not using Tagalong categories and are not use Store Pages, then Drop Locations can remove tens- or even -hundreds-of-thousands of locations in less than a minute.    By comparison a 200,000 location table can take as long as 2 hours using Clear Locations.

Clear Locations

Clear Locations is a more thorough removal mechanism.  This process leaves the data tables intact but deletes the records one-at-time using the WordPress data interface which invokes all the hooks-and-filters.  That means any third party or CSA-derived add-on packs will fire all the “location cleanup” processes including Tagalong data cleanup , Store Pages cleanup, and any related meta data that may be hiding in WordPress options tables, taxonomy tables, custom page types, or custom data tables installed by add-on packs.

Janitor Change Log

Posted on

Analyzing WordPress PHP Memory Consumption

XDebug Banner

This weekend I have been processing a large 200,000 location data file for a Store Locator Plus customer.   This is one of the larger files I have processed on my test system and it is the first file over 60,000 locations I’ve processed since Store Locator Plus 4.2 and WordPress 4.x have been released.    This large file processing and the geocoding required is taxing several systems in the Store Locator Plus hierarchy.  WordPress, Google OEM API calls, and the locator are all showing their weak spots with this volume of data processing.   They can all handle it to some degree, but maximizing efficiency is the key.

The temporary solution to most of the issues is to increase memory and process limits.   These are some of the key findings, as posted on the CSV Import documentation pages for Store Locator Plus:

Check your php.ini post_max_size setting if doing a direct file import versus a cron URL based import. post_max_size is typically set to 8 (MiB) on most servers.   This is typically enough for around 25,000 locations but it depends on how long your descriptions are and how many data fields you have filled out.   SLP 4.2.41 will warn you if you try to upload a file larger than your post_max_size limit.

Check your php.ini memory_limit setting and make sure it is large enough to handle the WordPress overhead plus the size of your CSV file times two.   The WordPress database interface and the CSV file processing will consume lots of memory.  The more plugins, widgets, and advanced theme features you have more more memory WordPress will use and the more PHP memory will leak over time. A setting of 256M is enough for approximately 15,000 locations.

Check your wp-config WP_MEMORY_LIMIT.   You may need to add this define to wp-config.php.  define(‘WP_MEMORY_LIMIT’ , ‘256M’).  The number needs to be equal-to or less-than the php.ini memory-limit.    It is the WordPress-specific memory limit and works with php.ini memory_limit.

Check your wp-config WP_MAX_MEMORY_LIMIT.   You may need to add this define to wp-config.php.  define(‘WP_MAX_MEMORY_LIMIT’ , ‘256M’).   This is the WordPress admin interface memory limit and works like WP_MEMORY_LIMIT for admin pages.

Set Duplicates Handling to Add especially if you know you do not have duplicate locations in your data.  SLP 4.2.41 further improves the performance when using ‘add’ mode by eliminating extra data reads from the database.

Set Server-To-Server speed to Fast under the General Settings tab unless you are on a shared host or experience a large number of uncoded locations during import.

Set the PHP Time Limit to 0 (unlimited) under the General Settings tab.   For hosting providers that allow your web apps to change this, the unlimited value will let the import run to completion.

Keep in mind Google limits you to 2500 latitude/longitude (geocoding) lookups per 24 hours per server IP address.  If you are on a shared host you share that limit with all other sites on that host.

However, even with all of these settings tweaked to fairly high values for my VirtualBox development system running on a MacBook Pro Retina host, the 4GB of RAM allocated to WordPress still is not enough.   The system eventually runs out of memory when the file gets close to the 45,000 location mark.  Luckily the “skip duplicate addresses” option allows the process to continue.    The “out of memory” error still rears its ugly head in the wpdb  WordPress database engine and is a problem for handling larger files.

Enter Xdebug and memory profiling.   Somewhere buried in the Store Locator Plus code, WordPress code, PHP MySQL interface, or PHP core engine there is a memory leak.  With a complex application environment finding the leak is going to be a monumental task.  It may not be something I can fix, but if I can mitigate the memory usage when processing large files that will help enterprise-class sites use Store Locator Plus with confidence.

Getting Xdebug On CentOS 7

If you follow my blog posts on development you will know that I run a self-contained WordPress development environment.  The system uses Vagrant to fire up a VirtualBox guest that runs CentOS 7 with GUI tools along with a full WordPress install including my plugin code.   This gives me a 2GB “box file” that I can ship around and have my full self-contained development environment on any system capable of running VirutalBox.   Here is how I get Xdebug connected to my local Apache server running WordPress.

Install xdebug from the yum install script.


# sudo yum install php-pecl-xdebug.x86_64

Turn on xdebug in the php.ini file


# find / -name xdebug.so

/usr/lib64/php/modules/xdebug.so

#sudo vim /etc/php.ini

zend_extension="/usr/lib64/php/modules/xdebug.so"

Check if xdebug is installed:


# php --version

... PHP 5.4.16
.... with xdebug v2.2.7

Enable some xdebug features by editing php.ini again.

Read about XDebug Profiling.

Read about XDebug Tracing.


# sudo vim /etc/php.ini

xdebug.default_enable=1  ; turns on xdebug any time a PHP page loads on this local server

xdebug.idekey="PHPSTORM" ; in case I turn on the automated listener for built-in PHP Storm debugging/tracing

xdebug.profiler_enable = 1 ; turn on the profiler which creates cachegrind files for stack trace/CPU execution analysis

xdebug.profiler_enable_trigger = 1;  turn on a cookie "hook" so third party browser plugins can turn the profiler on/off with a bookmark link

xdebug.profiler_output_dir = "/var/www/xdebug" ; make sure this directory is writable by apache and readable by your local user

xdebug.auto_trace = 1 ; when any page loads, enable the trace output for capturing memory data

xdebug.show_mem_delta = 1 ; this is what tells trace to trace memory consumption changes on each function call

xdebug.trace_output_dir = "/var/www/xdebug" ; same idea as the profiler output, this will be where trace txt files go

Restart the web server to get the php.ini settings in effect:


# sudo service httpd restart

At this point I can now open any WordPress page including the admin pages.   Shortly after the page has rendered the web server will finish the processing through xdebug and a trace* file will appear in /var/www/xdebug.   I can now see the stack trace of the functions that were called within WordPress with the memory consumption at each call.     This is the start of tracking down which processes are eating up RAM while loading a large CSV file without adding thousands of debugging output lines in the web app.

Be warned, if you are tracing large repetitive processes your trace file can be many GiB in size, make sure you have the disk space to run a full trace.

Posted on

Social Media and Events Add On Updates

phpUnit Test ELM Banner

Social Media Extender and Events Location Manager have been updated for Store Locator Plus today.

Both updates addressed an issue with display the social media icons in the results output.   Text labels can now be shown under each of the icons for these add-on packs.

Change Log

 

 

Posted on

Show Your States Widget Update

CSA News and Info Banner

The Widget Pack for Store Locator Plus has been updated with a new SLP States widget.   The new widget places a drop down menu showing all of the states where you have locations along with a submit button so your users can easily see where your are located.    When the user selects a state and clicks the submit button your Store Locator Plus map page will be brought up showing all non-private locations within that state.

As part of this update the SLP Search Form widget has also been updated.   The most visible change is the MAP URL field is not a single entry that takes the full URL to your Store Locator Plus map page.  If you are not using SEO friendly permalinks and have ?page_id=8 in your URL, for example, the widget pack will sort that out for you automatically.    The search form also enables immediate results mode, fixing an issue on sites that did not want an immediate list of locations to appear on the map page but did want them to appear when a user came in from a widget search.    The search form also drops the number of locations limit and now shows all locations within the defined radius.

 

Changelog

Posted on

Locator Patch Finds Missing Locations

Twitter 1500x500 Map 3D Banner

Store Locator Plus 4.2.38 was released today addressing an issue where some sites suddenly “lost” locations.

After a week of research it was discovered that some MySQL and MariaDB installations will set boolean data fields to NULL versus true/false or blank.    This led to an issue where any of the locations that had a null value in the privacy flag entry would be considered private by the Store Locator Plus search mechanism.  The new patch considers locations with a NULL privacy value to not be marked private, which has “recovered” missing locations on the front-end search interface.

Enhancements

In addition to the patch, several updates that were underway have been included as part of the effort to launch a Widget Pack update and start provide more advanced user interface features and WordPress theme support.

SLP 4.2.38 Widget Tagalong Pro Pack
Version 4.2.38 with Widget Pack, Tagalong, and Pro Pack options activated.

A new plugin theme has been bundled with Store Locator Plus that has been test for layout compatibility with the iThemes Herschel WordPress theme.

SLP 4.2.38 iThemes Herschel Plugin Theme
SLP 4.2.38 iThemes Herschel Plugin Theme with Pro Pack, Enhanced Search, and Enhanced Results.

The Simple White Four Column theme has been updated to use the newer SaSS based CSS rules engine, patching some quirks that are addressed with the core Store Locator Plus CSS ruleset included in all SaSS based plugin themes.

SLP 4.2.38 Simple White 4 Column on WP theme iThemes Herschel
WordPress iThemes Herschel theme with Store Locator Plus Simple 4 Col White theme. Uses Pro Pack, Enhanced Search, Enhanced Results for layout control.

A new general layout option for Pro Pack users allows site designers to gain access to not just the location data but also Store Locator Plus plugin option values.   The new [[slp_option nojs=”<option_name>”]] and [[slp_option js=”<option_name>”]] settings allow site designers to do things like place headers on the map with dynamic messages that change when site options change such as “All distances shown are in <miles>” where miles will change to kilometers if you change the default measurement in the UX settings panel.     Many other options such as a fixed default radius and various labels can be displayed using this feature.     It is used in conjunction with Enhanced Search to display heading on search box labels that can be changed via the admin panel (coming in the next Enhanced Search release).

The language translation system was extended, providing better internationalization (i18n) and localization (l10n) support for all add-on packs.  This has been built into the SLP 4.2 add-on framework.

The state filter processor was refined to eliminate redundant code in Enhanced Search and the upcoming Widget Pack update.

The state and country SQL processors were updated to address various null data issues and to provide slightly faster SQL query processing.

Map Center and Zoom Level have been migrated to the newer options system in Store Locator Plus. This also grants access to the slp_option shortcode in Pro pack for displaying these values such as “Distances calculated from <map center address>.” when a map is first loaded.

Change Log