Ask your WordPress questions! Pay money and get answers fast!

Make wp_generate_attachment_metadata fast WordPress

SOLVED

Hello,

we are using Gravity forms for user submitted content. The problem is that the submission of the form is extremely slow because on submission, the wp_generate_attachment_metadata function is carried out and all image sizes are produced and all other information are gathered. This can take up to 1 minute for large images. So we would like to have an alternative to the following code that make large image upload more stable (up to 10MBs per image) and submission way faster. Many thanks!

//Multi upload field - Insert images into WP media library and link them to post ID
add_filter("gform_after_submission_20", "my_add_post_attachements",10,2); //Your form ID
function my_add_post_attachements($entry, $form){

$submit = array();
$submit = $entry["38"]; //The field ID of the multi upload
$submit = stripslashes($submit);
$submit = json_decode($submit, true);

foreach ($submit as $key => $value){

// Get the path to the upload directory.
$wp_upload_dir = wp_upload_dir();

// Image URL
$url = $value;

// Full PATH to image in Upload Directory
$filename = basename( $url );
$file = $wp_upload_dir['path'] . "/$filename";

// Full PATH to image in Upload Directory
$url2 = $wp_upload_dir['url'] . "/$filename";

// The ID of the post this attachment is for.
$post_id = $entry["post_id"];

// Check the type of tile. We'll use this as the 'post_mime_type'.
$filetype = wp_check_filetype( $file );

// Prepare an array of post data for the attachment.
$attachment = array(
'guid' => $url2,
'post_mime_type' => $filetype['type'],
'post_title' => preg_replace( '/\.[^.]+$/', '', $filename ),
'post_content' => '',
'post_parent' => $post_id,
);

// Make sure that this file is included, as wp_generate_attachment_metadata() depends on it.
require_once( ABSPATH . 'wp-admin/includes/image.php' );

// Insert the attachment.
$attach_id = wp_insert_attachment( $attachment, $file, $post_id );

if ( !is_wp_error($attach_id) ) {
// Generate the metadata for the attachment, and update the database record.
$attach_data = wp_generate_attachment_metadata( $attach_id, $file );
wp_update_attachment_metadata( $attach_id, $attach_data );
}
}
}

2014-07-08 by robork

Answers (4)

2014-07-08

John Cotton answers:

Why not remove the meta generation from the loop altogether and then have a cron job running that does the work in the background after the response has been passed to the user?

You wouldn't want to use a straight WP Cron task as that would just move the delay along to another unsuspecting user.

But you could configure a page or custom url that gets called by an external cron service and that triggers the work.

In your loop you would just store (transient API?) the id of the attachment that needs processing (effectively building up an array of these until the external cron runs).

The cron would retrieve the current array, process each id, discard each as it went.

robork comments:

Hi John,

this really sounds like a desirable solution to us. I have some knowledge in php but this seems to go beyond my knowledge. I wouldn't really know how to do this. Could you help out at this point?

Many thanks!

John Cotton comments:

Alter the end of your current code to:
if ( !is_wp_error($attach_id) ) { update_post_meta( $attach_id, '_meta_required', $file ); }

Then create a function somewhere like this:
function process_meta() { foreach ( get_posts( array( 'post_type' => 'attachment', 'meta_key' => '_meta_required', 'posts_per_page' => -1 ) as $post ) { $file = get_post_meta( $post->ID, '_meta_required', true ); $attach_data = wp_generate_attachment_metadata( $post->ID, $file ); wp_update_attachment_metadata( $post->ID, $attach_data ); delete_post_meta( $post->ID, '_meta_required' ); } } if( isset($_GET['process_meta']) ) { process_meta(); exit(0); }

Just call http://your_site.com?process_meta=true to run the code. You can set up a cron on your own server or use an external service like webcron.org.

You should add security and some error handling, but that's the basic idea.

robork comments:

Hi John,

I have added the code to the function php (also added the bracket to the second part: foreach ( get_posts( array('post_type' => 'attachment', 'meta_key' => '_meta_required', 'posts_per_page' => -1 )) as $post ) { )

Now I don't have thumbnails and I get an error message in the edit post section even though I have called
http://our_site.com?process_meta=true

This parameter is not really doing anything I suppose as this is just loading our page normally.

Do you have any idea how to work this out? Ideally we would like the processing to take place automatically in the background after the post has been created in the database (after submission) without using external services and having security issues. Do you think we could work something out here?

Maybe provide your email and we can provide access.

Many thanks!

robork comments:

Something along the lines of this code where the thumbnails are only created when actually needed:

//Add additional image sizes only when needed
add_filter('image_downsize', 'ml_media_downsize', 10, 3);
function ml_media_downsize($out, $id, $size) {
// If image size exists let WP serve it like normally
$imagedata = wp_get_attachment_metadata($id);
if (is_array($imagedata) && isset($imagedata['sizes'][$size]))
return false;

// Check that the requested size exists, or abort
global $_wp_additional_image_sizes;
if (!isset($_wp_additional_image_sizes[$size]))
return false;

// Make the new thumb
if (!$resized = image_make_intermediate_size(
get_attached_file($id),
$_wp_additional_image_sizes[$size]['width'],
$_wp_additional_image_sizes[$size]['height'],
$_wp_additional_image_sizes[$size]['crop']
))
return false;

// Save image meta, or WP can't see that the thumb exists now
$imagedata['sizes'][$size] = $resized;
wp_update_attachment_metadata($id, $imagedata);

// Return the array for displaying the resized image
$att_url = wp_get_attachment_url($id);
return array(dirname($att_url) . '/' . $resized['file'], $resized['width'], $resized['height'], true);
}

John Cotton comments:

<blockquote>Now I don't have thumbnails and I get an error message in the edit post section even though I have called
http://our_site.com?process_meta=true</blockquote>

Have you looked in the post meta table to see if the meta field has been created?

<blockquote>This parameter is not really doing anything I suppose as this is just loading our page normally.</blockquote>
Are you sure a caching plugin isn't hiding output?

<blockquote>Ideally we would like the processing to take place automatically in the background after the post has been created in the database</blockquote>
Unless you trigger a separate process as I have suggested, then the user has to bear the brunt of the processing time.

robork comments:

Hi John,

1) yes, it seems that the meta field has not been created.

2) yes, I'm sure

3) Is the async POST curl a solution to this?

Let me know what you think I can do.

thanks!

John Cotton comments:

<blockquote>1) yes, it seems that the meta field has not been created.</blockquote>
Can you post the code you have so that I can see exactly what you've got? Until that's fixed none of the rest will work.

robork comments:

Functions.php:

//Multi upload field - Insert images into WP media library and link them to post ID
add_filter("gform_after_submission_20", "my_add_post_attachements",10,2); //Your form ID
function my_add_post_attachements($entry, $form){

$submit = array();
$submit = $entry["38"]; //The field ID of the multi upload
$submit = stripslashes($submit);
$submit = json_decode($submit, true);

foreach ($submit as $key => $value){

// Get the path to the upload directory.
$wp_upload_dir = wp_upload_dir();

// Image URL
$url = $value;

// Full PATH to image in Upload Directory
$filename = basename( $url );
$file = $wp_upload_dir['path'] . "/$filename";

// Full PATH to image in Upload Directory
$url2 = $wp_upload_dir['url'] . "/$filename";

// The ID of the post this attachment is for.
$post_id = $entry["post_id"];

// Check the type of tile. We'll use this as the 'post_mime_type'.
$filetype = wp_check_filetype( $file );

// Prepare an array of post data for the attachment.
$attachment = array(
'guid' => $url2,
'post_mime_type' => $filetype['type'],
'post_title' => preg_replace( '/\.[^.]+$/', '', $filename ),
'post_content' => '',
'post_parent' => $post_id,
);

// Make sure that this file is included, as wp_generate_attachment_metadata() depends on it.
require_once( ABSPATH . 'wp-admin/includes/image.php' );

// Insert the attachment.
$attach_id = wp_insert_attachment( $attachment, $file, $post_id );

if ( !is_wp_error($attach_id) ) {

update_post_meta( $attach_id, '_meta_required', $file );

}

// if ( !is_wp_error($attach_id) ) {
// // Generate the metadata for the attachment, and update the database record.
// $attach_data = wp_generate_attachment_metadata( $attach_id, $file );
// wp_update_attachment_metadata( $attach_id, $attach_data );
// }
}
}

function process_meta() {

foreach ( get_posts( array('post_type' => 'attachment', 'meta_key' => '_meta_required', 'posts_per_page' => -1 )) as $post ) {

$file = get_post_meta( $post->ID, '_meta_required', true );

$attach_data = wp_generate_attachment_metadata( $post->ID, $file );

wp_update_attachment_metadata( $post->ID, $attach_data );

delete_post_meta( $post->ID, '_meta_required' );

}

}

if( isset($_GET['process_meta']) ) {

process_meta();

exit(0);

}

John Cotton comments:

There are a few superfluous lines in there but nothing I can see that would stop the meta value being created.

Are you sure that the meta value isn't in the database?

robork comments:

Not from what I can see - maybe tell me where to look to double-check that Im looking for the right value.

John Cotton comments:

You should be looking in the wp_postmeta table for rows with a meta_key of '_meta_required'

2014-07-08

Dbranes answers:

You might check out the [[LINK href="http://www.php.net/manual/en/book.pthreads.php"]]pthreads[[/LINK]] of PHP.

I've not tested it yet, but it looks promising:

<blockquote>pthreads is an Object Orientated API that allows user-land multi-threading in PHP. It includes all the tools you need to create multi-threaded applications targeted at the Web or the Console. PHP applications can create, read, write, execute and synchronize with Threads, Workers and Threaded objects.
</blockquote>

Here's another idea:

You could try out "async" curl requests within your gform_after_submission_20 hook,
to send the data to another script that handles the meta data generation.

This might be of interest:

[[LINK href="https://github.com/vrana/php-async"]]https://github.com/vrana/php-async[[/LINK]]
[[LINK href="https://github.com/vrana/php-async/blob/master/CurlAsync.php"]]https://github.com/vrana/php-async/blob/master/CurlAsync.php[[/LINK]]

Some people are using proc_open() or even fsockopen (here's one [[LINK href="http://blog.markturansky.com/archives/205"]]example[[/LINK]]).

If you want to play with async POST curl you could try this demo function and modify the parameters to your needs:

function demo_async_curl_post( $url = '',$postfields = array() ) { $curl = curl_init(); curl_setopt( $curl, CURLOPT_URL, $url); curl_setopt( $curl, CURLOPT_POST, TRUE); curl_setopt( $curl, CURLOPT_POSTFIELDS, $postfields ); curl_setopt( $curl, CURLOPT_USERAGENT, 'api'); curl_setopt( $curl, CURLOPT_TIMEOUT, 1); curl_setopt( $curl, CURLOPT_HEADER, 0); curl_setopt( $curl, CURLOPT_RETURNTRANSFER, false); curl_setopt( $curl, CURLOPT_FORBID_REUSE, true ); curl_setopt( $curl, CURLOPT_CONNECTTIMEOUT, 1 ); curl_setopt( $curl, CURLOPT_DNS_CACHE_TIMEOUT, 10 ); curl_setopt( $curl, CURLOPT_FRESH_CONNECT, true ); $data = curl_exec($curl); //echo $data; curl_close($curl); }

Dbranes comments:

Here's how you can use the above demo function, to send an async curl POST request to the slow running script called heavy.php from your hook:

add_filter( "gform_after_submission_20", "my_add_post_attachements",10,2 ); function my_add_post_attachements( $entry, $form ){ // ... if( function_exists( 'demo_async_curl_post' ) { demo_async_curl_post( 'http://example.com/heavy.php', array( 'foo' => 'bar' ) ); } // ... }

where

function demo_async_curl_post( $url = '',$postfields = array() ) { $curl = curl_init(); curl_setopt( $curl, CURLOPT_URL, $url); curl_setopt( $curl, CURLOPT_POST, TRUE); curl_setopt( $curl, CURLOPT_POSTFIELDS, $postfields ); curl_setopt( $curl, CURLOPT_USERAGENT, 'demo'); curl_setopt( $curl, CURLOPT_TIMEOUT, 1 ); $data = curl_exec($curl); curl_close($curl); }

robork comments:

Hi Dbranes,

I'm not a pro in php and really don't know how to use this with our code and in our example. Could you be a bit more specific or try to give an example for our code?

Thanks!

Dbranes comments:

I created (untested) demo for you on github:

[[LINK href="https://gist.github.com/dbranes/042ec65fec4a991c6381"]]https://gist.github.com/dbranes/042ec65fec4a991c6381[[/LINK]]

where you have to make few modifications to match your setup.

robork comments:

Oh thanks a lot!

If I copy the heavy.php I get a syntax error in line 27&28 where it says:

if( $mysecret != $input_secret
|| empty $input_url )
|| empty $input_post_id )
)

Is this just a demo or will it actually be something we could employ long-term?

Dbranes comments:

ahh, I updated the gist.

Hopefully better now.

robork comments:

Yes syntax error is gone and I have put the script online. I have adapted all the parts that needed to include info from our website. But unfortunately now success. Any idea what I'm missing out? Do I have to include a library? Here is the code:

functions.php:
//Multi upload field - Insert images into WP media library and link them to post ID

add_filter( 'gform_after_submission_20', 'my_add_post_attachements', 10, 2 );

function my_add_post_attachements($entry, $form){

$submit = array();
$submit = $entry["38"]; //The field ID of the multi upload
$submit = stripslashes($submit);
$submit = json_decode($submit, true);

// Edit this to your needs:
$my_secret = 'hackbraten';
$my_heavy_script_url = 'http://robork.de/wp-content/themes/robork/heavy.php';

// Loop:
if( function_exists( 'demo_async_curl_post' ) )
{
foreach ( $submit as $key => $value )
{
$post_fields = array(
'url' => $value,
'post_id' => $entry['post_id'],
'key' => $my_secret
);

demo_async_curl_post( $my_heavy_script_url, $post_fields );
}
}

}

if( function_exists( 'demo_async_curl_post' ) )
{
function demo_async_curl_post( $url = '',$post_fields = array() )
{
$curl = curl_init();
curl_setopt( $curl, CURLOPT_URL, $url);
curl_setopt( $curl, CURLOPT_POST, TRUE);
curl_setopt( $curl, CURLOPT_POSTFIELDS, $postfields );
curl_setopt( $curl, CURLOPT_USERAGENT, 'demo');
curl_setopt( $curl, CURLOPT_TIMEOUT, 1 );
$data = curl_exec( $curl );
curl_close( $curl );
}
}

heavy.php:
<?php

/**
* heavy.php
*
* Script for generating meta data from POST inputs ( url, post_id, key )
*
* version 0.0.1
*/

// Include WordPress:
define( 'WP_USE_THEMES', false );
require( './wp-blog-header.php' ); // Edit this path to your needs

// Secret key:
$my_secret = 'hackbraten'; // Edit this to your needs.

// POST input parameters:

$input_post_id = filter_input( INPUT_POST, 'post_id', FILTER_SANITIZE_NUMBER_INT );
$input_url = filter_input( INPUT_POST, 'url', FILTER_SANITIZE_URL );
$input_secret = filter_input( INPUT_POST, 'secret', FILTER_SANITIZE_NUMBER_INT );

// Validate input:
if( $mysecret != $input_secret
|| empty ( $input_url )
|| empty ( $input_post_id )
)
{
die('No Access');
}

// Your script:

// Get the path to the upload directory.
$wp_upload_dir = wp_upload_dir();

// Image URL
$url = $input_url;

// Full PATH to image in Upload Directory
$filename = basename( $url );
$file = $wp_upload_dir['path'] . "/$filename";

// Full PATH to image in Upload Directory
$url2 = $wp_upload_dir['url'] . "/$filename";

// The ID of the post this attachment is for.
$post_id = $input_post_id;

// Check the type of tile. We'll use this as the 'post_mime_type'.
$filetype = wp_check_filetype( $file );

// Prepare an array of post data for the attachment.
$attachment = array(
'guid' => $url2,
'post_mime_type' => $filetype['type'],
'post_title' => preg_replace( '/\.[^.]+$/', '', $filename ),
'post_content' => '',
'post_parent' => $post_id,
);

// Make sure that this file is included, as wp_generate_attachment_metadata() depends on it.
require_once( ABSPATH . 'wp-admin/includes/image.php' );

// Insert the attachment.
$attach_id = wp_insert_attachment( $attachment, $file, $post_id );

if ( !is_wp_error($attach_id) ) {
// Generate the metadata for the attachment, and update the database record.
$attach_data = wp_generate_attachment_metadata( $attach_id, $file );
wp_update_attachment_metadata( $attach_id, $attach_data );
}

Dbranes comments:

You must also edit this part:

require( './wp-blog-header.php' ); // Edit this path to your needs

to match the relative or absolute path to the wp-blog-header.php file in your WP root directory.

robork comments:

Oh I see. I have changed it to:

require( ABSPATH . 'wp-blog-header.php' ); // Edit this path to your needs

Still no success. Anything else I need to change/consider/add?

Many thanks!

Dbranes comments:

aha, you can't use ABSPATH there, because you haven't included WordPress yet ;-)

robork comments:

require( 'http://robork.de/wp-blog-header.php' ); // Edit this path to your needs

Do I need to include wordpress in this file?

Dbranes comments:

ps: notice that this must be directory path, not url.

robork comments:

Well, I give it another try:
require( '/is/htdocs/wp1025602_ND5ABK6JFF/robork/wp-blog-header.php' ); // Edit this path to your needs

Dbranes comments:

yes or relative from where it's included, something like:

require_once( './../../../wp-blog-header.php' );

but after your last edit, the heavy.php script seems to be running now, I get "No Access" when I view it directly.

robork comments:

Ok so that seems to be working. However, I don't think the script is running in the background – at least I dont get the expected results.

Would it be possible that I provide access for you and you give it a try? Or what do you think is the best way to proceed?
Many thanks!

robork comments:

Would be awesome. What do I need to provide or do you want to check the code again?

robork comments:

Would be awesome. What do I need to provide or do you want to check the code again?

2014-07-08

Sébastien | French WordpressDesigner answers:

why do you not limit the authorized size of uploaded images ?

robork comments:

We do, but even 2MB per image is too large and makes the process slow and even cancel.

Sébastien | French WordpressDesigner comments:

maybe could you try this plugin http://wordpress.org/plugins/front-end-upload/

robork comments:

We are using Gravity Forms Multi image Upload so we won't be able to integrate this. Thanks though!

2014-07-08

timDesain Nanang answers:

this code will remove all add_image_size, except medium size.
result: will produce 2 images only (full and medium size)

put this code in the theme's functions.php

add_action('wp', 'wpq_wp', 1); function wpq_wp(){ global $post; if(is_page( 157 )) // change with your page id add_filter( 'intermediate_image_sizes', 'wpq_new_sizes', 99 ); } function wpq_new_sizes( $sizes ){ $all_sizes = $sizes; unset($sizes); $new_sizes = array(); foreach($all_sizes as $key=>$val){ if($key=='medium') // change with yours $new_sizes[$key] = $val; } return $new_sizes; }

robork comments:

Thanks Tim,

I have played around with removing the image sizes but the submission is still very slow without producing the different image sizes. I suspect the wp_generate_attachment_metadata function does even some more than creating thumbnails.

timDesain Nanang comments:

i think wp_generate_attachment_metadata or wp_update_attachment_metadata based on add_image_size.

Defined at: wp-admin/includes/image.php, line 89 as get_intermediate_image_sizes function

timDesain Nanang comments:

hmm, you can check the value of _wp_attachment_metadata in the prefix_postmeta table
then compare the value after tweaking and before