News & Info

Daily Updates and Tech Chatter

ThemeForest and “Texturizing” Shortcodes

I just spent that past 2 weeks trying to hunt down a bug in Store Locator Plus 3.6.    After a few weeks of reports of “open HTML” appearing on pages I finally had enough reports of the problem to realize there was a common denominator.   I had GUESSED that the problem was buried deep in my plugin, trigger some sort of Localization or Internationalization issue within WordPress.   I came to that conclusion because the broken HTML was using the #8217 code in place of single apostrophes I used within my HTML code.  That is a WordPress specific thing.

Turns out the problem is a bug in a basic function that many ThemeForest theme developers appear to have copied from one-another.   ThemeForest themes often use the wptreats_formatter function to add some “Magic Sauce” to post contents.  Unfortunately they are breaking several rules and best practices when doing this.  To understand what is going on & why you first need to know a bit about WordPress and how content is rendered.   For the sake of keeping a potentially long article as short as possible I will skip all the details and focus on only the relevant parts.

WordPress Manhandles Your Content

First, we need to understand some WordPress “trickery” that goes on with your posts and pages.   By default, when you create a page or post your main content is “texturized”.  This is a process by which WordPress uses their filters system and changes your content whether your like it or not.   Most themes don’t change this behavior and thus some interesting formatting happens.   For example, if you simply press enter when creating your form post, WordPress will translate that to an HTML <br/> code, ensuring your post appears with the proper line breaks as you typed it in your post.    If you type a single apostrophe (‘), WordPress changes it to the non-standard &#8217 escape sequence.  That renders as a custom character which is similar to an apostrophe visually but is actually, in simple English, a “long backtick” character.

Themes and Plugins Manage Your Content

Themes and plugins can do various things with your content.  One of those things is to render a shortcode.  For example, when you type

search provided by Store Locator Plus
Enter Your Address or Zip Code Above.
in your page or post, the SLP search form and map appear.   We take that square-bracket code and intercept it.  We then create custom in-post content that is primarily HTML with some JavaScript and graphics mixed in.  With very little work on your end we create a lot of content in your page or post.

Normally we can output basic HTML into your page or post by having our code return basic HTML.  It typically gets into the page unchanged because the standard "texturizer" in WordPress, the wptexturize() function, does NOT manhandle properly formatted HTML tags.   It leaves them along so an entry like this:

<table

cellpadding='2'

>

blah...

 

Remains just like that, whitespace, line breaks and all.

 

Telling WordPress "DO NOT TOUCH"

Sometimes a plugin can render code that is too complex for the wptexturize() function to handle properly.  In these rare cases you can tell WordPress not to touch your output when rendering HTML via shortcodes or script output.   They have a special filter that says "don't even THINK ABOUT changing my content when you see the STORE-LOCATOR shortcode".

With SLP3.6.1 the code was updated under the assumption that the problem was with Store Locator Plus itself and how it interacted with WordPress.  We used the filters and told WordPress explicitly, do not change my stuff.   It works great.

 

Usually.

ThemeForest Ignores The Rules

After writing the patches and running some tests, it looked like the new patch may work.  However many of the people that reported the open HTML problem were  still having issues.    When the 5th report came in it became obvious it was not a plugin conflict but a theme issue.   All 5 reports used a different theme but they ALL were from ThemeForest.

I bit the bullet and purchased the $40 Natural Theme from ThemeForest.  Within minutes I saw the problem.   An hour later I found the source of the issue.   All 5 themes used a special function called "wptreats_formatter()".   All 5 themes also disabled the standard WordPress wptexturize() filter applied to content.

What ThemeForest does is a bit evil and probably unintentional.  First of all they call wptexturize() within wptreats_formatter(), so it almost works the same way.  To a casual observer the results will look the same 90% of the time.  However there are some differences.  wptreats_formatter() is called AFTER all content, shortcodes, and scripts are called.   That means the built-in WordPress "ignore this please, don't touch" filters are useless.  The shortcodes are already rendered and no longer appear as shortcodes, making it impossible for wptexturize() to even tell that a shortcode was in place.    Post-shortcode rendering is NOT when wptexturize() should be called.

Secondly, wptreats_formatter() is breaking up the content into "pieces" and thus causing valid HTML to be split.  This secondary feature of the wptexturize() function, "do not texturize valid html" is now broken as well.   The <table...> tag in my example is now split up.  wptexurize() now sees this as open HTML and says "this is incomplete/invalid HTML, it must be part of written content and not HTML instructions" and then texturizes it.

The ThemeForest setup has a  process bug where the ThemeForest wptreats_formatter() is not adhering to the proper call ordering for the function.  The setup calls wptexturize() post shortcode rendering. That is not the right time for this call.   The shortcode filter is rendered useless as this point.

The ThemeForest setup has a secondary processing bug that is splitting content into segments without regard to complete context of the content.   This is breaking up valid HTML segments and hands wptexturize() pieces of an article content that are not whole.   Where it splits can cause partial HTML to go into wptexturize().  This is not the proper way to pass content to wptexturize().

Summary

It is the combination of these two process bugs that has made Store Locator Plus version 3.6 appear to be broken on the many sites that use Theme Forest themes.   The SLP 3.6 code was updated to make the HTML more readable by putting the cellpadding and style HTML attributes for the table tag on a new line.   This triggered a deeply buried series of bugs in the ThemeForest wptreats_formatter() function.

Luckily wptreats_formatter() has a built-in "skip our hacking of your content" tool of its own.  The ThemeForest system looks for the [raw]...[/raw] shortcodes around any string.   We are using this loophole with version 3.6.2 ONLY if we detect the wptreats_formatter() function is active.   Adding a random [raw] shortcode to our output is not the preferred way to handle this, but that is what makes our plugin work for ThemeForest customers.  Anyone that is using a theme that does not have this function will not get the extra [raw] wrapper and hopefully will not see any side effects from this patch.

Now to tell ThemeForest of this issue and see if they have a response and/or issue a patch to all their affected themes.

Tags: , , ,

1 Awesome Comments So Far

Don't be a stranger, join the discussion by leaving your own comment
  1. lcleveland
    November 1, 2012 at 3:53 PM #

    Envato / Theme Forest has responded. The notable parts of their response…

    “We’re well aware of the issue, but it’s been problematic to find a solution for our authors as well as stopping them from perpetuating this very bad practice.”

    “If it’s acceptable for your customers, then you’d be welcome to let them know that you’ll contact us and we’ll get the theme updated for them to resolve the issue. Unfortunately though, this will actually be ultimately in the hands of the theme author, and there’s potential that if it’s an old theme the author may simply prefer to take it off the marketplace.

    It is a tough situation, but I’m happy to work with you in any way I can.”

    So I asked for a copy of the problematic themes so I could find a workaround without having to shell out $50 for every broken theme to fix THEIR bugs. The response…

    “Unfortunately we can’t do this. You need a license to use the themes, and that can only be obtained by purchasing.”

    So I try again, and tell them … “That’s unfortunate, I don’t really want/need the themes. I’m just trying to fix the HTML and related bugs reported by users with specific themes.”

    To which the answer makes it obvious Envato knows there is a problem but really doesn’t care that much about fixing it. According to Envato “… it’s a support issue for the developer of that theme. Adding the theme to your unsupported list is just what you’ll have to do until the issue is resolved…”

    So that answers that question. While Envato was giving a glimmer of hope that they may actually provide decent customer service they once again have failed to deliver.