Reason #10 for choosing TYPO3: ExtBase + Scheduler = WIN

This post goes out to @dstankowski who asked me to follow up on the ExtBase talk I gave at the 2011 San Francisco TYPO3 conference. During my presentation, I mentioned that we had written an importer task that relied heavily on ExtBase, although I didn't go into too much detail about how it worked. Time is short today, so I'm going to keep this post as brief as I can. If I forget to cover anything, let me know in the comments and I'll try to clarify.

The work that this post describes was done for a client that needed to display project records on the front-end of their site. These projects are fairly complex records, with a handful of many-to-many relations to categorization records (constituents, services provided, category, geographic scope, etc). There are probably 15-20 fields on the project record, in addition to five or six relational fields and a geocoding component. A big part of the project was creating a front-end interface for searching the projects based on seven or eight criteria, as well as a single view for when users drill down to a single project and a map view. To get all this to work, we spent some time developing a fairly robust project model, as well as models for the various supporting records.

One twist on this project, however, was the fact that the project records did not live in TYPO3. The client's project intake process took place in Salesforce, where projects were represented by Salesforce Account objects. In addition to building out the models and the UI, our job was to provide a mechanism to pull project data from Salesforce into TYPO3 every hour or so using Salesforce's web services. We've been down this road many times, and we usually end up writing import scripts, which are mostly one-offs, to get the job done. Those import scripts tend to contain logic for handling database updates and, to some degree, for working with the various data models. In this case, budget was tight and we wanted to reduce code redundancy, so we set about finding a way for our import mechanism to take advantage of the ExtBase models that we'd already developed as well as the code we'd written for each model's storage repository. Why write a bunch of database queries in the import script if we could take advantage of existing ExtBase classes? The first step was setting up a scheduler task. Now that we have a scheduler in TYPO3, we rarely write scripts that use TYPO3's CLI interface anymore (did you know TYPO3 has a CLI interface? Well, it does!), as it's easier to just write tasks. When we built this site (maybe 6 months ago), there was some hackery involved in utilizing ExtBase within a scheduled task. This hackery may no longer be necessary; I haven't gone back to look at this code in relation to recent versions of ExtBase so I'm not entirely sure.

Ok, so, first thing first. Define the import task in your ext_localconf.php file:

And don't forget to autoload the classes we'll need in your ext_autoload.php file:

You can see the aforementioned hackery in this file. Because of limitations with how ExtBase is initialized, I needed to break out the actual logic from a class that instantiates it, which is why we're including both the projectImport script and the projectImportLogic script. This technique was based, as I recall, on a post on the ExtBase newsgroup, although I can't recall the exact post. If you find it, let us know in the comments. The additionalFieldProvider class is used on this task to provide a field where the user can set some additional configuration on the task, such as storage pid and email.

Next, we setup the project import class. This class extends the tx_scheduler_Task class because, if I remember correctly, all tasks must extend that class. However, because we're using ExtBase for our class, we'll store the actual import logic in the ProjectImportLogic class. This task class is essentially just a wrapper around actual import. Our class contains some code for setting the Salesforce WSDL, figuring out the storage pid, and whatnot. You may not need that logic in your task. The important part is the execute method, which instantiates and executes our importLogic object.

Now, create the importLogic class. This class will extend the ExtBase bootstrap. There may be a better way to do this now, and extending the bootstrap probably doesn't fall under the category of "stable API," so your mileage may vary here. The basic class looks like this:

Next, we setup the execute method:

A lot of this code is specific to what our task was doing, but it gives you a sense of how we approached it. Notice that we call "setupFramework" before we do the import. The setup framework initializes the ExtBase framework. As I've said above, there may be better ways to do this. If you know of one, post a comment and I'll update this post.

Once the ExtBase framework is initialized, we can use standard methods to setup repositories, instantiate objects, etc. For example:

Next, we wrote the code that pulled the records in from Salesforce. That code isn't relevant to this post, as it's specific to this client's domain. The actual doImport method is rather long, and it's where the magic happens. Instead of writing database queries to insert or replace projects in the TYPO3 database, we instantiate a project model and use its getters and setters. The advantage of using setters on the model is that we don't have to duplicate validation logic on the model and in the import script. We get greater code reusability, which leads to lower-costs for developing the extension (good for us, good for our client). Instead of managing many-to-many relationships and doing database queries, you just update your models and add objects to your repositories. The end result is that your import script can focus on its domain -- mapping objects from one system to another -- instead of infrastructure (database queries, property validation, etc).

You can see this focus on the correct domain in the actual import method:

When I have time to follow up on this, perhaps sometime next week, I'll write another post that looks at how we're approaching these data import mechanisms in FLOW3. We're currently building an internal FLOW3 application that interfaces with an accounting system, a time tracking system, and other external services. In this application, there's a lot of bidirectional syncing of data between the FLOW3 app and other services. To handle this, we're trying to use FLOW3's property mapper as much as possible, which has the potential (perhaps not yet fully realized in its current implementation) to make these mapping operations easier and more automated.

Let me know what you think. The above code examples are not intended to be cut and paste into other extensions, so keep that in mind. Rather, they're meant to illustrate a general approach and to point folks in the right direction as they try to use extBase in conjunction with TYPO3's scheduler.

Was this helpful? Want more blog posts? Follow me on twitter and keep the discussion going!


Michael McManus (@momcmanus) just pointed out to me that the FLOW3 CLI is being backported to ExtBase: When that's done, maybe we'll go back from executing these sorts of things from the CLI. I've been working with the FLOW3 CLI and have been pretty impressed.
Posted by Zach Davis on September 7, 2011

Bravo - a handy walk-through, and an excellent real-world case of how the model-based paradigm can reduce development time and make application code more easily understandable at a glance.
Posted by Gabe Blair on September 8, 2011

Thank you for sharing your concept. We have applied it successfully in our project, but a question has come up: How do we deal with the importation of multi-language records? How do we localize an object in Extbase?
Posted by Dominik Stankowski on September 14, 2011

I found that we can set $l18nParent after we have added and persisted the records in the default language. So the multi-language issue has been solved. Really nice script now.
Posted by Dominik Stankowski on September 19, 2011

@Dominik - That's great! I'm glad you guys were able to figure out a solution. Hopefully this will help others in the future.
Posted by Zach Davis on September 19, 2011

Thanks a lot for this example!
I have question though,
what is this for:
'tx_extkey_task_pidadditionalfieldprovider' => t3lib_extMgm::extPath('extkey', 'Classes/Task/PidAdditionalFieldProvider.php'),

and what do you actually write in this class ?

Posted by Miguel on November 4, 2011

I second Gabe's perspective - very handy indeed! I'll certainly have to put this to the test.
Posted by Company Culture on November 7, 2011

Has anyone tested this in TYPO3 v4.6 (extbase 1.4) already?
Posted by Kilian on November 28, 2011

Great blog you have here.. It's hard to find quality writing like yours nowadays.
I honestly appreciate individuals like you! Take care!!
Posted by Boucher on November 18, 2013

Leave A Comment

Cast Iron Coding, 107 SE Washington, Suite 210, Portland, OR 97214
PH: 503.841.5669 | FAX: 866.285.6140 |

Made in Oregon!

Cast Iron Coding is a web development studio located in beautiful Portland, Oregon

Contact Us
107 SE Washington, Suite 210
Portland, OR 97214
PH: 503.841.5669
FAX: 866.285.6140