Using WP-All Import, Repeat Upload process for paged variable? WordPress

• SOLVED

Hello,

I'm using WP All Import to import items to a custom post type from an API response. This is all working great, with the exception that the API only delivers a max of 25 responses at a time. To get around this, I've currently set up different import processes for each known page (i.e. .../events/?page=3 in my "download from url").

This is obviously very tedious because the # of pages from the API continues to grow. Can anyone think of a way in which I could leverage the total pages response and have the import run for all subsequent urls automatically every time its run?

The response body starts as such. Note the next which is meant to help navigate through:

{
"total_pages": 5,
"per_page": 25,
"page": 1,
"total_records": 120,
"_links": {
"next": {
"href": "https://somedomain.com/api/v2/event_campaigns/eb84c205-7865-43dc-9e75-3479bf694c26/events?page=2"
},


I'll also note that I currently have function to pass needed headers, and this is also very inefficient as could not think of a way to wildcard the domain, and so the if statement repeats for each domain:

add_action( 'http_api_curl', 'my_curl_header', 10, 3 );
function my_curl_header( $handle, $r, $url ) {
if ( $url == 'https://somedomain.com/api/v2/event_campaigns/eb84c205-7865-43dc-9e75-3479bf694c26/events/' ) {
curl_setopt( $handle, CURLOPT_HTTPHEADER, array(
'OSDI-API-Token: APIKEYHERE',
'Content-Type: application/json'
) );
}

Answers (2)

2020-02-02

John Cotton answers:


$url = 'https://somedomain.com/api/v2/event_campaigns/eb84c205-7865-43dc-9e75-3479bf694c26/events';
while( $url ) {
$response = wp_remote_get( $url );

// Process rest of response here

if( isset( $response->_links ) && isset( $response->_links->next ) ) {
$url = $response->_links->next->href;
} else {
$url = null;
}
}


---

John Cotton comments:

This assumes that "next" is not present rather than blank or empty when there are no more records. But you could just add an empty() test in the if statement if that was the case.

---

Andrew Clemente comments:

Thanks, and sorry for the delay as I had also emailed the WP all import guys to see if they had any advice.

They pointed me towards this dynamic url function. How could I combine the above to cycle through these (and yes, the next parameter is not present on the last page)

Thank you!

---

Andrew Clemente comments:

https://gist.github.com/mbissett/d9a202abfd5c45400f09ad581755b30f

---

John Cotton comments:

You don't need a "dynamic url", the next value gives you the correct one each time.

Have you tried my code? Pretty sure it will do what you want.

2020-02-05

Shabeer M answers:

Please use the following recursive function for use the dynamicaly creasting page url


function import_posts($url='') {
$response = wp_remote_get( $url );
if ( is_wp_error( $response ) ) {
$error_message = $response->get_error_message();
echo "Something went wrong: $error_message";
} else {
/*
* Run the import function/code if any
*/

if(isset($response->_links) && isset($response->_links->next) {
import_posts($response->_links->next->href);
} else {
return;
}
}
}


here is the function is import_posts(), and this will check the next URL is exist on the response, then the function call again itself.